Multi-scale converter for image analysis

Abstract

The invention provides a method for processing an image and an image processing system. The method comprises: processing, by one or more processors, a set of images comprising a native resolution image and one or more resized variants of the native resolution image, encoding a corresponding spatial embedding of each image block associated with a respective region of a given one of the native resolution image or the one or more resized variants to form a set of spatially encoded image blocks; applying, by the one or more processors, the scale embedded set to the set of spatially encoded image blocks to capture scale information associated with a given one of the native resolution image and the one or more resized variants to form a set of input markers; and creating, by the one or more processors, a final image representation via a self-attention process on the set of input markers.

Description

[0001] This case is a divisional application of the invention patent application with application date of July 1, 2021, application number 202180100108.6, and invention name “Multi-scale transformer for image analysis”. Background Art

[0002] Image analysis is used for many different types of applications. Some types of image analysis, such as image quality assessment (IQA), can quantify the perceived quality of an image. Other types of image analysis can be used to classify aspects of an image, such as objects that appear within an image. Such analysis can include analyzing images using a convolutional neural network (CNN). However, such methods may be limited by fixed-size input requirements for training models with small batches. Resizing or cropping an image to fit a given size (with an associated aspect ratio) may affect image synthesis, introduce distortion or other artifacts, and thus change the quality of the image. Attempts to solve such technical problems may raise other issues, such as requiring large amounts of data storage, requiring resizing and intelligent grouping, or generally making training on large datasets impractical. Summary of the invention

[0003] Aspects of the technology employ a patch-based multi-scale transformer that can be used for IQA and other imaging applications. This approach, discussed in detail below, avoids the restriction of fixed input size of images and effectively predicts the quality of native resolution images. The method converts native resolution images into multi-scale representations, allowing the transformer's self-attention mechanism to capture information of both fine-grained detailed patches and coarse-grained global patches. Self-attention can focus on the entire input sequence, so it can effectively capture image quality at different granularities. A hash-based two-dimensional spatial embedding is adopted, which maps patch locations to a fixed grid to effectively handle images with arbitrary aspect ratios and resolutions. In addition, since the patch locations at each scale are hashed to the same grid, it aligns spatially close patches at different scales, allowing the transformer model to exploit information at multiple scales. In addition to the spatial embedding, an independent scale embedding is introduced to help the transformer distinguish patches of different scales in the multi-scale representation. Various embodiments of the disclosed technology can solve technical problems such as inefficient image analysis due to the need to change the aspect ratio and / or resolution of the input image. The disclosed method can also enable better image analysis on a wider range of different images (i.e., not just images that already conform to a specific aspect ratio).

[0004] According to one aspect of the technology, a method for processing an image is provided. The method includes: constructing, by one or more processors, a multi-scale representation of a native resolution image, the multi-scale representation comprising the native resolution image and a set of resized variants that preserve aspect ratios; encoding, by one or more processors, a corresponding spatial embedding of each image block associated with a corresponding region of one of the sets of resized variants that preserve aspect ratios, thereby forming a set of spatially encoded image blocks; applying, by one or more processors, the set of scale embeddings to the set of spatially encoded image blocks to capture scale information associated with the native resolution image and the set of resized variants of aspect ratios, thereby forming a set of input tokens; and performing, by one or more processors, self-attention on the set of input tokens according to a transformer encoder module to create a final image representation.

[0005] In one example, the final image representation corresponds to a predicted quality score of the native resolution image. Constructing the multi-scale representation may include segmenting each of the native resolution image and each aspect ratio-preserving resized variant into fixed-size image patches, wherein each image patch represents a different region of one of the native resolution image or the aspect ratio-preserving resized variant.

[0006] In another example, encoding the corresponding spatial embedding includes hashing the image patch position of each image patch within the grid of the learnable embedding. In a further example, each aspect ratio preserving resized variant is formed such that the aspect ratio of the variant is resized according to the longer side of the native resolution image. The multi-scale representation may include a native resolution image having a height H, a width W, and one or more channels C, and a set of aspect ratio preserving resized variants is derived using a Gaussian kernel, wherein each channel represents a color component of the native resolution image.

[0007] In yet another example, to align the set of spatially encoded image patches across scales, image patch positions at all scales are mapped to the same grid. In a further example, the method further includes: prepending a learnable classification tag to the set of input tags before self-attention.

[0008] In another example, performing self-attention on a set of input tokens includes performing multi-head self-attention on the set of input tokens, including: performing a first normalization on the set of input tokens to generate a first normalized set; performing a multi-head self-attention operation on the first normalized set; performing a second normalization on the output of the multi-head self-attention operation to generate a second normalization; and performing a multi-layer perceptron processing on the second normalization to create a final image representation. In this case, the second normalization can be performed on the output of the multi-head self-attention operation of the residual connection added to the set of input tokens. The final image representation can be obtained by adding the second normalized input to the output of the multi-layer perceptron processing.

[0009] In yet another example, the method further includes pre-training a transformer of the transformer encoder module on a selected image dataset. The size of the image patch can be selected based on an average resolution across a set of native resolution images and resized variants that preserve aspect ratio. Here, the size of the image patch can be further based on a size of a grid of learnable embeddings to which the image patch is mapped.

[0010] According to another aspect of the technology, an image processing system includes a memory configured to store an image and one or more processors operably coupled to the memory. The one or more processors are configured to: construct a multi-scale representation of a native resolution image, the multi-scale representation comprising the native resolution image and a set of resized variants that preserve aspect ratios; encode a corresponding spatial embedding of each image block associated with a corresponding region of one of the native resolution image or the set of resized variants that preserve aspect ratios, thereby forming a set of spatially encoded image blocks; apply the set of scale embeddings to the set of spatially encoded image blocks to capture scale information associated with the native resolution image and the set of resized variants of aspect ratios, thereby forming a set of input tags; and perform self-attention on the set of input tags using a transformer encoder module to create a final image representation.

[0011] In one example, the processor is further configured to store at least one of a final image representation, a native resolution image, or a set of aspect ratio-preserving resized variants in a memory. The final image representation may correspond to a predicted quality score for the native resolution image. Constructing the multi-scale representation may include segmenting each of the native resolution image and each aspect ratio-preserving resized variant into fixed-size image patches, wherein each image patch represents a different region of the native resolution image or one of the aspect ratio-preserving resized variants. Encoding the corresponding spatial embedding includes hashing the image patch position of each image patch within a grid of learnable embeddings. The one or more processors may also be configured to prepend a learnable classification tag to the set of input tags prior to self-attention.

[0012] In yet another example, performing self-attention on a set of input tokens includes performing multi-head self-attention on the set of input tokens, including: performing a first normalization on the set of input tokens to generate a first normalized set; performing a multi-head self-attention operation on the first normalized set; performing a second normalization on the output of the multi-head self-attention operation to generate a second normalization; and performing a multi-layer perceptron processing on the second normalization to create a final image representation.

[0013] According to another aspect of the technology, a method for processing an image is provided, the method comprising: encoding, by one or more processors, a set of images including a native resolution image and one or more resized variants of the native resolution image, a corresponding spatial embedding of each image block associated with a corresponding region of a given one of the native resolution image or the one or more resized variants to form a set of spatially encoded image blocks; applying, by one or more processors, a set of scale embeddings to the set of spatially encoded image blocks to capture scale information associated with the native resolution image and the given one of the one or more resized variants to form a set of input tags; and creating, by one or more processors, a final image representation via a self-attention process on the set of input tags.

[0014] According to another aspect of the technology, an image processing system includes: a memory configured to store an image; and one or more processors operably coupled to the memory, the one or more processors configured to: encode a set of images including a native resolution image and one or more resized variants of the native resolution image, and a corresponding spatial embedding of each image block associated with a corresponding region of a given one of the native resolution image or the one or more resized variants to form a set of spatially encoded image blocks; apply a set of scale embeddings to the set of spatially encoded image blocks to capture scale information associated with the native resolution image and the given one of the one or more resized variants to form a set of input tags; and create a final image representation via a self-attention process on the set of input tags. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 An example CNN-based model using cropped or resized images is shown.

[0016] Figure 2 Example multi-scale image representations in accordance with aspects of the present technique are shown.

[0017] Figure 3 An example model overview of a multi-scale transformer for image quality assessment in accordance with aspects of the present technique is shown.

[0018] Figure 4A-4C Multi-resolution image representations with preserved aspect ratios in accordance with aspects of the present technique are shown.

[0019] Figure 4D-Figure 4G The model formation stages according to aspects of the present technique are shown.

[0020] Figure 5A An example transformer encoder in accordance with aspects of the present technique is shown.

[0021] Figure 5B Features of multi-head self-attention according to various aspects of the present technology are illustrated graphically.

[0022] Figure 6 Examples of attention on original and resized images of a multi-scale composition are shown in accordance with aspects of the present technique.

[0023] Figure 7A-7C Shown is a visualization of learned HSE cosine similarities in accordance with aspects of the present technique.

[0024] Figure 8A-8D Visualizations of each image in a multi-scale representation in accordance with aspects of the present technique are shown.

[0025] Figure 9A-9B A system is shown for use with various aspects of the present technology.

[0026] Fig.10 Methods according to aspects of the present technology are shown. DETAILED DESCRIPTION

[0027] Overview

[0028] The patch-based multi-scale transformer is configured to construct a multi-scale image representation as input. This representation includes a collection of native resolution images and resized variants that preserve aspect ratio (ARP). As discussed further below, each image is segmented into fixed-size patches that are embedded by a patch encoding module. To capture the two-dimensional structure of the image and handle images of different aspect ratios, the spatial embedding is encoded by hashing the patch positions within a grid that can be learned. Scale embeddings are introduced to capture scale information. The transformer encoder receives input tokens and performs multi-head self-attention. The output can be predicted by adding a classification token to the sequence to represent the entire multi-scale input and using the corresponding transformer output as the final representation.

[0029] Although the specification is described primarily with reference to image quality analysis (IQA), it should be understood that the disclosed technology is not limited to such applications, but has broad applicability. For example, applications in which the technology can be used include IQA, image classification, image representation learning where the task requires learning representations of images, and multimodal learning (both text and images). In an embodiment where a patch-based multi-scale transformer is trained to predict quality scores, the predicted quality scores can be further used for applications such as image sorting, image recommendation, and bad image filtering. The technology is also applicable to subtasks in IQA, such as estimating some technical properties of an image (e.g., brightness, contrast, compression rate). Image classification using the methods discussed herein can involve, for example, general object classification tasks, as well as classification in biology or medicine where the image size should not be adjusted.

[0030] By being able to process images in their native aspect ratio, the transformers described herein can achieve superior results on a wide range of image analysis tasks when compared to existing systems (e.g., as measured by metrics related to the accuracy of the transformer output).

[0031] Figure 1 An example method 100 is shown in which a CNN-based model is used for image quality assessment. As shown, in a CNN-based model, a full-size input image 102 needs to be resized or cropped to a fixed shape 104 for batch training. However, such preprocessing may change the image aspect ratio and / or composition, which may adversely affect image quality. Figure 2 As shown in the view 200 of FIG. 1 , the image patch-based multi-scale transformer method can process the full-size image 202 into detailed views and generate resized images 2041, . . . , 204 k Extract multi-scale features from a collection of

[0032] Overall architecture

[0033] Figure 3 A diagram 300 of a model overview of a multi-scale transformer according to aspects of the present technique is provided, which can be used, for example, for image quality assessment. As shown in 302, a multi-scale image representation is constructed as an input to the transformer encoder. The representation includes a native resolution image (304) and its ARP resized variants (3061, ..., 306 k ). In this example, each image 304 and 306 is segmented into fixed-size patches 308, which are embedded by a patch encoding module 310 to form a multi-scale patch embedding set 312. In order to capture the 2D structure of the image and handle images of different aspect ratios, the patch position (i, j) is hashed to (t i ,t j) to encode the spatial embedding, as shown in the G xG array 314, and the resulting set of embeddings 316 is shown above the multi-scale image patch embedding set 312. A set of scale embeddings 318 is introduced to capture scale information. The transformer encoder module 320 takes the resulting input token set and performs multi-head self-attention. In order to obtain an output for the image (e.g., a prediction of image quality or a classification of the image), a classification token [CLS] 322 can be applied to the sequence to represent the entire multi-scale input. The corresponding transformer output can be used as a final representation associated with the native resolution image, such as a quality score or classification (not shown). Various aspects of the model are described in detail below.

[0034] More specifically, a multi-scale representation of the input image is created, which contains the native resolution image and its ARP resized variants. Figure 4A As shown in example 400, there are two resizing variants, although there may be three, four, or more resizing variants. The resizing variants may be formed such that the aspect ratio is resized according to the longer side of the original image (i.e., the longer side is fixed to L_i as shown).

[0035] Images of different scales are divided into image blocks of fixed size and fed into the model. Figure 4B A view 410 is shown, where image patches 412 are displayed as different boxes per image. Since the image patches are from images of different resolutions, the system effectively encodes the multi-aspect ratio multi-scale input into a sequence of labels ( Figure 3 The image block encoding module (blocks 312, 316 and 318) captures pixel, spatial and scale information. Figure 3 310 in ) encodes the image patches extracted from the multi-scale representation. The hash-based spatial embedding module encodes the 2D spatial position of each image patch. And the learnable scale embedding encodes the different scales. After encoding the multi-scale input into a labeled sequence, the system can preset additional learnable "classification tags" [CLS] (see Figure 3 322). In this case, the CLS tag state at the output of the transformer encoder is used as the final image representation. A fully connected layer can be added to predict the image quality score. The tag is a vector of fixed size. The final input tag includes scale, spatial information, and image patch information.

[0036] Multi-scale image patch embedding

[0037] Image quality can be affected by both local details and global composition. To capture both global and local information, the technique models the input image using a multi-scale representation. Image patches of different scales enable the transformer to aggregate information at multiple scales and spatial locations. In one example, the multi-scale input consists of a full-size image with height H, width W, channels C, and a sequence of ARP-resized images of the full-size image using a Gaussian kernel. Each channel represents a color component of the image. For example, when the input image is in RGB format, it has 3 channels. Grayscale images have only 1 channel, and RGBA images have 4 channels (RGB+alpha). For RGB images, each pixel is represented by three [0,255] values ​​of pixel intensity, each representing the R channel, G channel, and B channel value, respectively. Therefore, an RGB image of height H and width W can be represented by a matrix of shape [H,W,3].

[0038] Here, the resized image has height h k , width w k and channels C, where k = 1,...,K, where K is the total number of resized variants of each input. To align the resized images to obtain a consistent global view, for each resized variant, the longer side length is fixed to L k . Resize factor α k Can be defined as:

[0039] α k = L k / max (H,W), where h k = α k H, w k = α k W (1)

[0040] Depending on a scenario, there may be no restrictions on how to frame the resizing variants, as long as the length is consistent during training and testing.

[0041] Figure 4C A view 420 is shown showing an embedded "crop" (image patch) 422 of the original resolution image and crops 4241 and 4242 of the resized image (K=2). These form the image patch pixel embedding part of the model. Once preprocessing is complete, the crops are input to the image patch encoding module ( Figure 3 310 in ).

[0042] In the multi-scale representation, square image patches (crops) of size P are extracted from each image. For example, the image patch size P can be 16x16, 32x32, 64x64, or larger or smaller sizes. For images whose width or height is not a multiple of P, the image can be padded with zeros accordingly. The image patches are encoded by the image patch encoding module to encode each image patch into a D-dimensional embedding. Here, D is the potential tag size used in the transformer. Although the image patches can be encoded using linear projection, in one case, a 5-layer ResNet type convolutional neural network with a fully connected layer of size D can be used as the image patch encoder module to learn a better representation of the input image patches. When pre-trained on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) such as the ILSVRC-2012 ImageNet, encoding the image patches using several convolutional layers (e.g., up to 5 layers) may perform better than linear projection. Since the image patch encoding module is lightweight and shared across all input image patches of size P, it only adds a small number of parameters. For example, the table below shows an example of the total number of parameters for models with different image patch encoding modules using the AVA dataset. Here, for a 32x32 image patch size, 2 layers of convolution add 1 million (1M) parameters relative to linear projection, while 5 layers of ResNet add 5 million (5M) parameters. For example, Spearman rank ordered correlation (SRCC), Pearson linear correlation (PLCC), 2 layers of convolution can use the root of ResNet (7x7 convolution followed by 3x3 convolution). For a 5 layer ResNet, residual blocks are stacked on top of 2 layers of (simple) convolution.

[0043] #parameter SRCC PLCC Linear Projection 22M 0.703 0.716 2 layers of convolution 23M 0.717 0.731 5-layer ResNet 27M 0.726 0.738

[0044] Table 0: Comparison of different image block encoding modules on the AVA dataset

[0045] The sequence of image block embedding outputs of the image block encoding module is concatenated together to form a multi-scale embedding sequence of the input image. The number of image blocks for the original image and the resized image can be calculated as N = HW = P respectively. 2 and n k =h k w k / P 2 . Therefore, the number of image patches may be different for each image.

[0046] Since each input image has a different resolution and aspect ratio, H and W are different for each input, so N and nk Different. To obtain fixed-length inputs during training, the encoded image patch tokens can be zero-padded or clipped to the same length. An input mask can be attached to indicate valid inputs, which will be used in the transformer to perform masked self-attention. Note that the padding operation does not change the inputs, as they are ignored by masking the padded tokens in the multi-head attention.

[0047] As mentioned above, for each resizing variant, the longer side length is fixed to L k .therefore, and the system can be safely filled to m k For native resolution images, the system can simply pad or crop the sequence to a fixed length l. Since the sequence length can be arbitrary, padding is not necessary during single-input evaluation.

[0048] Hashing-based 2D spatial embedding

[0049] Spatial position embeddings are useful for injecting awareness of 2D image structure into 1D sequence inputs. Fixed-length position embeddings specify an embedding for each input position; however, this fails for variable input resolutions where the number of patches varies, so each patch in the sequence could be from an arbitrary position in the image. Furthermore, independently localized position embedding models cannot align spatially compact patches from different scales. In light of this, aspects of the present technique are configured to meet the following requirements. First, effectively encode patch spatial information at different aspect ratios and input resolutions. Second, spatially compact patches of different scales should have compact spatial embeddings. Third, the embedding should be efficient and easy to implement, and non-intrusive to transformer attention. Here, non-intrusive means that the system does not require changes to the transformer attention implementation, as opposed to relative position embedding methods that require changes to the transformer attention portion.

[0050] In view of the above, a hashing-based 2D spatial embedding (HSE) is proposed, where the image patch at row i and column j is hashed to the corresponding element in a G x G grid (see Figure 3 314 in ). Figure 4D The diagram 430 in FIG. 4 shows how input crops of image patches are hashed or otherwise mapped to corresponding elements in a G x G grid. For example, hashing may be implemented using a lookup table.

[0051] Each element in the grid is a D-dimensional embedding. HSE consists of a learnable matrix Definition. Assume the input resolution is H x W. The input image will be divided into Image block. For an image block at position (i, j), its spatial embedding is given by the position (t i ,t j) where:

[0052]

[0053] like Figure 3 As shown, embed the D-dimensional space into Add to the image patch embedding element-wise. Figure 4E View 440 shows the spatial embeddings of the cropped image block pixel embeddings 442, 4441, and 4442 applied to each resolution.

[0054] For fast lookups, the system can simply replace (t i t j ) is rounded to the nearest integer. HSE does not require any changes to the Transformer Attention Module. In addition, t i and t j The computation and lookup of are lightweight and easy to implement without requiring unnecessary computing resources.

[0055] To align patches across scales, patch locations at all scales are mapped to the same grid T. Therefore, patches located close together in the image but at different scales will be mapped to a tight spatial embedding in T, since i and H as well as j and W change proportionally with the resizing factor α. This achieves spatial alignment across different images represented at multiple scales.

[0056] There is a trade-off between expressiveness and trainability for choosing the hash grid size G. Smaller G (e.g., 4 or less) may result in too many collisions between patches, which makes it difficult for the model to distinguish between spatially tight patches. Larger G (e.g., 100 or more) may consume a lot of memory resources and may require more different resolutions for training. Larger G means that the hashes are sparser, so more different resolutions are required for training, otherwise some locations may not have enough data to learn good representations. For learnable T, a good rule of thumb is to let the grid size multiplied by the number of patches P be roughly equal to the average resolution, so G x G x P x P = H x W. For different datasets with an average resolution of about 450x 500 and a patch size of 32, the grid size may be between 10 and 15. In one scenario, based on an IQA setting where coarse position information is sufficient, it was found that once G is large enough, changing G only leads to small performance differences. Therefore, in one example, G can be between 5-15, such as 10, although larger or smaller values ​​of G can be used.

[0057] Scale Embedding

[0058] Since the same hash matrix can be reused for all images, HSE does not distinguish between image patches of different scales. Therefore, additional scale embedding (SCE) is introduced to help the model effectively distinguish information of different scales and better utilize cross-scale information. In other words, SCE marks which input scale the image patch comes from in the multi-scale representation. Figure 4F The view 450 of 454 shows the scale embeddings 452, 4541, and 4542 applied to the spatial embeddings 442, 4441, and 4442 and the cropped image patch pixel embeddings 442, 4441, and 4442 for each resolution. As described above, in order to obtain a fixed length input during training, the encoded image patch labels can be zero-padded to the same length. This is in Figure 4G As shown in view 460 of , zero padding blocks 462 , 4641 , and 4642 are used.

[0059] SCE is defined as a learnable scale embedding of an input image with K scale resizing variants After spatial embedding, the first element is added element-wise to all D-dimensional image patch embeddings of the native resolution image. is also added element-wise to all image patch embeddings of the image resized at scale k.

[0060] Pre-training and fine-tuning

[0061] In one example, a transformer model is pre-trained on a large dataset (e.g., a dataset with a million or more images, such as ImageNet) and fine-tuned on a downstream task. During pre-training, the system may keep random crops as augmentations to generate images of different sizes. However, the system may skip resizing and initialize the model for inputs with different resolutions and aspect ratios instead of doing square resizing. Augmentations such as RandAugment and mixup can be used in pre-training. For example, for the augmentation operation pool, RandAugment randomly selects N from the pool and applies it sequentially with magnitude M (N, M are configurable). Mixup is a data augmentation technique that generates weighted combinations of random image pairs from the training data. Other suitable pre-training datasets include ImageNet21k, JFT-300M, and OpenImage.

[0062] When fine-tuning on IQA tasks, the system does not need to resize or crop the input images to maintain image composition and aspect ratio, so that the model has an unmodified input image at the beginning of processing. In one scenario, the only augmentation used in fine-tuning is random horizontal flipping. Here, for each training image, the system can randomly flip it horizontally with probability p, where p can be 0.5 (or more or less). For evaluation, the method can be directly applied to the original image without aggregating multiple augmentations (e.g., multi-crop sampling).

[0063] When fine-tuning on an IQA dataset (which may include images labeled with corresponding IQA measurements / scores), regression losses such as the L1 loss for a single mean opinion score (MOS) and the Earth Mover Distance (EMD) loss can be used to predict the quality score distribution:

[0064]

[0065] Where p is the normalized score distribution and CDF p (m) is the cumulative distribution function The EMD loss penalizes the distribution distance between the rating and the prediction. Here, r is a hyperparameter that selects the r-norm distribution distance. For example, r can be set to 2 to penalize the Euclidean distance between CDFs.

[0066] When the system is trained to perform other image analysis tasks, other training data sets and loss functions (or objective functions) may be used. For example, for a classification task, a classification data set including multiple image inputs and corresponding labels representing specific categories of images may be used to train the transformer. A suitable loss function such as a cross entropy loss may be used to train the transformer in such a scenario. It will be appreciated that, as previously described, the transformer may be used for other image analysis tasks, and any suitable data set and / or loss function may be used to train the transformer to perform an image analysis task.

[0067] Converter Encoder

[0068] Figure 5A An example transformer encoder 500 is shown in FIG. As shown, the embedded multi-scale representation 502 is input to a transformation encoder block 504. The transformation encoder block 504 includes the following layers: a first layer normalization 506, a multi-head self-attention (MSA) layer 508, a second layer normalization 510, and a multi-layer perceptron (MLP) layer 512. As shown by arrows 514 and 516, residual connections are added between the layers.

[0069] Multi-scale image patches are encoded as Where k=0...K is the scale index and n is the patch index in the scale. k=0 represents the full-size image. K is an adjustable parameter. The hash-based 2D spatial embedding (HSE) and scale embedding (SCE) are added to the patch embedding to form a multi-scale representation input. The learnable [class] tag embedding can be pre-set in the embedded tag sequence (x class ). The converter encoder can be formulated as:

[0070]

[0071] z0=[x class ; E p +E HSE +E SCE ] (5)

[0072] z′ q =MSA(LN(z q -1))+z q-1 , q=1…L (6)

[0073] z q =MLP(LN(z′ q ))+z′ q , q=1…L (7)

[0074]

[0075] E p is the image patch embedding. E HSE and E SCE are spatial embedding and scale embedding respectively. l is the number of image blocks at the original resolution. m1...m K is the number of image patches of the resized variant. z0 is the input to the transformer encoder. g is the output of each transformer layer, and L is the total number of transformer layers.

[0076] Using a larger,l,in fine-tuning can improve the performance of the model. Since large-resolution images have more image patches than low-resolution images, when,is too small, some larger images may be truncated, thus degrading the model performance. For example,,l,can be chosen to be 128, 256, or 512 or more or less.

[0077] Combination Figure 5B Embodiment 520 discusses self-attention and its multi-head version, multi-head self-attention (MSA). Assume that the input sequence is composed of Represented by, vectors Q, K and V are its query (Q), key (K) and value (V) representations respectively. Q, K and V are respectively obtained by using the learnable matrix Generate by projecting the input sequence. h is the inner dimension of Q, K, V. Then use the attention weight Compute a weighted sum over V, where the attention weights are the pairwise similarities between Q and K.

[0078] Q=zU q , K=zU k , V=zU v (9)

[0079]

[0080] SA(z)=AV (11)

[0081] MSA is an extension of self-attention, where s self-attention operations (heads) are performed in parallel. The outputs of all heads are concatenated together and then used using a learnable matrix Projected to the final output. h can be set to D / s to keep the number of computations and parameters per s constant. This results in:

[0082] MSA(z)=[SA1(z);…;SA s (z)]U M (12)

[0083] In batch training, input masks are used to indicate valid inputs and padding tokens are ignored. Figure 5B As shown in block 526 in , a mask is added to the attention weights before performing the softmax function in block 528. Before the softmax step in Equation 10, the attention weights at invalid positions are close to zero by setting the corresponding elements to -inf (negative infinity). In one example, considering the mixed attention on the tags through all layers, the attention weights are averaged over all heads and then recursively multiplied. Therefore, Figure 5B = represents equations (10) and (11). MatMul block 522 represents the matrix multiplication (QK in equation 10) T ). The scale at block 524 is expressed in (Equation 10) according to The masking operation at block 526 sets the elements to be ignored to -inf (see equation 14 below). The Softmax block 528 The output of is normalized to a probability distribution. The final MatMul block 530 is the AV multiplication in Eq. 11.

[0084] The attention mask can be constructed as Among them: If attention

[0085]

[0086] Then, the masked self-attention weight matrix can be calculated as:

[0087]

[0088] Example Implementations

[0089] In one example, multi-scale representations are constructed as native resolution images and two ARP resized images are used, with L1=224 and L2. Therefore, the number of scales is K=3. The image patch size P=32. The dimension of the transformer input tokens is D=384, which is also the dimension of the pixel image patch embedding, HSE and SCE. The HSE grid size is set to G=10. A transformer with lightweight parameters (e.g., 384 hidden size, 14 layers, 1152MLP size and 6 heads) is used to make the model size comparable to ResNet-50. The total parameters of the final model are about 27 million.

[0090] In this example, the MST-IQA model is trained on ImageNet for 300 epochs using Adam with β1=0.9;β2=0.999, a batch size of 4096, weight decay of 0.1, and a cosine learning rate decayed from 0.001. In training, the maximum number of patches is set from 1 to 512 full-size images. For fine-tuning, SGD with momentum and cosine learning rates decayed from 0.0002, 0.002, 0.0001, 0.12 is used for 10, 30, 30, and 20 epochs on PaQ-2-PiQ, KonIQ-10k, SPAQ, and AVA, respectively. For AVA, the batch size is set to 512, for KonIQ-10k, the batch size is set to 96, and for the others, the batch size is set to 128. For AVA, we use the EMD loss with r=2. For the other datasets, L1 is used for the loss.

[0091] The model can be trained on one or more tensor processing units (TPUs). All results can be averaged over 10 runs. SRCC, PLCC and the corresponding standard deviation (std) are reported. Table 1 below shows the results of the PaQ-2-PiQ full-size validation and test sets, Figure 6View 600 graphically shows the attention of the original image and the resized image. The last row in the table of view 600 (full size, 384, 224) shows the test set results of Table 1. The image on the right side of the figure is a visualization of the attention of the output labels to multi-scale representations (original resolution image and two ARP resized variants). The images in this paper are resized to fit the grid, and the model input is 3 different resolutions. The model focuses on details in high-resolution images and focuses on the whole in low-resolution images. This shows that the model learns image quality at different granularity. Table 2 shows the results on the KonIQ-10k dataset. Table 3 shows the results on the SPAQ dataset. Table 4 shows the results on the AVA dataset.

[0092]

[0093] Table 1: PaQ-2-PiQ full-size validation and test sets

[0094] SRCC PLCC 0.909 0.923

[0095] Table 2: Results on the KonIQ-10k dataset

[0096] SRCC PLCC 0.917 0.921

[0097] Table 3: Results on the SPAQ dataset

[0098] Classification Accuracy Mean square error SRCC PLCC 0.815 0.242 0.726 0.738

[0099] Table 4: Results on the AVA dataset

[0100] Ablation can be used to evaluate the effectiveness of hash-based spatial embedding (HSE) and scale-based embedding (SCE). Table 5 below shows the ablation study of spatial embedding on AVA.

[0101] Spatial Embedding SRCC PLCC w / o embedding 0.704 0.716 Fixed length (without HSE) 0.707 0.722 HSE 0.726 0.738

[0102] Table 5: Ablation results of spatial embedding on AVA

[0103] For “Fixed Length (without HSE)”, the input is treated as a fixed-length sequence and a learnable embedding is assigned to each position. As shown here, there is a big gap between adding and not adding (“w / o embedding”) spatial embeddings. Even the fixed-length sequences are not as satisfactory as HSE. This is because of two reasons. First, the inputs have different aspect ratios. Therefore, each image patch in the sequence can come from a different position of the image. Fixed-position embeddings cannot capture this change. Second, since each position is modeled independently, there is no cross-scale information, which means that the model cannot locate spatially compact image patches of different scales in the multi-scale representation. In addition, fixed-length methods are inflexible because fixed-length spatial embeddings cannot be easily applied to large images with more image patches. In contrast, HSE makes sense in any case.

[0104] exist Fig. 7A View 700 (G=10), Figure 7B View 710 (G=5) and Figure 7C A visualization of the learned HSE cosine similarity is provided in view 720 (G=15) of . As shown in these views, HSEs in close spatial locations are more similar (lighter shades or colors, or brighter in the same general location), and it corresponds well to the 2D structure. For example, the bottom HSE is brightest at the bottom. This shows that HSE can effectively capture the 2D structure of the image.

[0105] Table 6 shows that adding SCE can further improve the performance compared with not adding SCE. This table shows that SCE helps the model capture scale information independently of spatial information.

[0106] Scale Embedding SRCC PLCC w / o embedding 0.717 0.729 With embedded 0.726 0.738

[0107] Table 6: Ablation results of AVA upscale embedding

[0108] Different designs can be used to encode image patches, including linear projections and a small number of convolutional layers. As shown in Table 7, using a simple convolution-based image patch encoding module can improve performance. For simple conv, the root of ResNet is used (7x7 conv, then 3x3 conv). For 5-layer ResNet, residual blocks are stacked on top of simple convolutions. Adding more convolutional layers may have diminishing returns, and it can be seen that a 5-layer ResNet can provide a satisfactory representation for image patches.

[0109] #parameter SRCC PLCC Linear Projection 22M 0.703 0.716 Simple convolution 23M 0.717 0.731 5-layer ResNet 27M 0.726 0.738

[0110] Table 7: Comparison of different image block encoding modules on the AVA dataset

[0111] An evaluation of the effect of patch size is shown in Table 8. Here, ablations are performed on different patch sizes P. From this set of results, it can be seen that patch size P = 32 has the most effective performance.

[0112]

[0113]

[0114] Table 8: Comparison of different image patch sizes on the AVA dataset

[0115] Fig.10 .Visualization of attention from output tokens to multi-scale representations. The “Input” column shows the input image. “Attn Full” shows the attention on the full-size image. “Attn L=384” and “Attn L=224” show the attention on the ARP resized images. Note that here the images are resized to fit the grid and the model input is 3 different resolutions.

[0116] To understand how the system uses self-attention to integrate information at different scales, we can visualize the average attention weights from the output token to each image for the multi-scale representation. Examples of this are shown in Figure 8A-8D 800, 810, 820, and 830 of FIG. In these examples, the attention weights are averaged over all heads and then recursively multiplied, taking into account the mixed attention between the tokens through all layers.

[0117] A TPU, CPU, or other computing architecture may be used to implement a patch-based multi-scale transformer according to the features disclosed herein. An example computing architecture is Fig.9A and Fig. 9B In particular, Fig.9A and Fig. 9B 902 and 904 are schematic diagrams and functional diagrams of an example system 900, respectively, which includes multiple computing devices and databases connected via a network. For example, computing device 902 can be a cloud-based server system. Databases 904, 906, and 908 can store, for example, original full-size images, multi-scale representations, and / or transformer modules (such as transformer encoding modules, image block encoding modules, etc.), respectively. The server system can access the database via network 910. The client device may include one or more of a desktop computer 912 and a laptop or tablet PC 914, for example, for providing original images and / or for viewing output visualizations.

[0118] like Fig. 9BAs shown, each of computing devices 902 and 912-914 can include one or more processors, memory, data and instructions. The memory stores information accessible by one or more processors, including instructions and data (e.g., models) that can be executed or otherwise used by the processor. The memory can be any type capable of storing information accessible to the processor, including computing device readable media. The memory is a non-temporary medium, such as a hard drive, a memory card, an optical disk, a solid state, etc. The system can include different combinations of the above, so that different parts of the instructions and data are stored on different types of media. The instruction can be a collection of any instructions that are directly executed by the processor (such as machine code) or indirectly executed (such as a script). For example, the instruction can be stored on a computing device readable medium as a computing device code. In this regard, the terms "instruction", "module" and "program" can be used interchangeably herein. The instruction can be stored in an object code format for direct processing by the processor, or stored in any other computing device language, including scripts or collections of independent source code modules that are interpreted on demand or pre-compiled.

[0119] The processor may be any conventional processor, such as a commercially available CPU, TPU, graphics processing unit (GPU), etc. Alternatively, each processor may be a dedicated device, such as an ASIC or other hardware-based processor. Fig. 9B The processors, memory, and other elements of a given computing device are functionally shown as being within the same block, but these devices may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, the memory may be a hard drive or other storage medium located in a housing different from that of the processor, such as in a cloud computing system located in server 902. Accordingly, references to a processor or computing device will be understood to include references to a collection of processors, computing devices, or memories that may or may not operate in parallel.

[0120] Input data, such as one or more original full-size images, can be operated by the transformer module to generate one or more multi-scale representations, image quality assessment data, etc. The client device can utilize this information in various applications or other programs to perform image ranking, image recommendation, image classification, and / or bad image filtering.

[0121] The computing device may include all components typically used in conjunction with a computing device such as the processor and memory described above, as well as a user interface subsystem for receiving input from a user and presenting information (e.g., text, images, and / or other graphical elements) to the user. The user interface subsystem may include one or more user inputs (e.g., at least one front face facing the camera (user), a mouse, a keyboard, a touch screen, and / or a microphone) and one or more display devices (e.g., a monitor with a screen or any other electrical device operable to display information (e.g., text, images, and / or other graphical elements)). Other output devices such as speakers may also provide information to the user.

[0122] User-related computing devices (e.g., 912-914) can communicate with a backend computing system (e.g., server 902) via one or more networks (such as network 910). Network 910 and intermediate nodes may include various configurations and protocols, including, for example, Bluetooth. TM Short-range communication protocol, Bluetooth LE TM , the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, and various combinations of the foregoing. Such communication may be facilitated by any device capable of sending data to and from other computing devices (such as modems and wireless interfaces).

[0123] In one example, computing device 902 may include one or more server computing devices having multiple computing devices, such as a load balancing server farm or a cloud computing system, which exchanges information with different nodes of a network for receiving, processing, and sending data from other computing devices. For example, computing device 902 may include one or more server computing devices capable of communicating with any one of computing devices 912-914 via network 910.

[0124] Image quality assessment information or other data derived from the transformer module, the module itself, the multi-scale representation, etc. can be shared by the server with one or more client computing devices. Alternatively or additionally, client devices can maintain their own databases, transformer modules, etc.

[0125] Fig.10A method 1000 for processing an image according to aspects of the technology is shown. The method includes: at block 1002, constructing a multi-scale representation of a native resolution image. The multi-scale representation includes the native resolution image and a set of resized variants that preserve aspect ratio. At block 1004, the method includes encoding a corresponding spatial embedding of each image block associated with a corresponding region of one of the native resolution image or the set of resized variants that preserve aspect ratio, thereby forming a set of spatially encoded image blocks. At block 1006, the method includes applying the set of scale embeddings to the set of spatially encoded image blocks to capture scale information associated with the native resolution image and the set of resized variants of aspect ratio, thereby forming a set of input tags. And at block 1008, the method includes performing self-attention on the set of input tags according to a transformer encoder module to create a final image representation.

[0126] The patch-based multi-scale transformer approach discussed in this paper is advantageous for many reasons and has wide applicability. For example, applications where this technique can be used include IQA, image classification, image representation learning where the task requires learning image representations, and multimodal learning (text and images). The predicted quality scores can be further used in applications such as image ranking, image recommendation, and bad image filtering. This technique is also applicable to subtasks in IQA, such as estimating some technical properties of an image (e.g., brightness, contrast, compression rate). Image classification using the approach discussed in this paper can involve, for example, general object classification tasks, as well as classification in biology or medicine, where images should not be resized. The system can represent text and images as tags and perform multimodal learning.

[0127] Although the technology herein has been described with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the technology. Therefore, it should be understood that many modifications may be made to the illustrative embodiments and other arrangements may be designed without departing from the spirit and scope of the technology as defined by the appended claims.

Claims

1. A method for processing an image, the method comprising: encoding, by one or more processors, a set of images including a native resolution image and one or more resized variants of the native resolution image, a corresponding spatial embedding of each image block associated with a corresponding region of a given one of the native resolution image or the one or more resized variants to form a set of spatially encoded image blocks; applying, by the one or more processors, a set of scale embeddings to a set of spatially encoded image patches to capture scale information associated with a given one of the native resolution image and the one or more resized variants to form a set of input tags; as well as A final image representation is created by the one or more processors via a self-attention process on the set of input tokens.

2. The method according to claim 1, wherein: The final image representation corresponds to the predicted quality score of the native resolution image.

3. The method of claim 1 , further comprising constructing a multi-scale representation of the native resolution image, the multi-scale representation comprising the native resolution image and the one or more resized variants.

4. The method according to claim 3, wherein: The one or more resizing variants form a set of aspect ratio preserving resizing variants.

5. The method according to claim 3, wherein: Constructing the multi-scale representation includes segmenting the native-resolution image and each resized variant into fixed-size image patches, where each image patch represents a region of one of the native-resolution image or the resized variant.

6. The method according to claim 3, wherein: The multi-scale representation includes a native-resolution image with height H, width W, and one or more channels C, and one or more resized variants are derived using a Gaussian kernel, where each channel represents a color component of the native-resolution image.

7. The method according to claim 1, wherein: Encoding the corresponding spatial embedding includes hashing the image patch position of each image patch within a grid where the embedding can be learned.

8. The method according to claim 1, wherein: Each resized variant is formed such that the aspect ratio of the variant is resized according to the longer side of the native resolution image.

9. The method of claim 1, further comprising aligning a set of spatially encoded image patches across scales by mapping image patch locations at all scales to a grid.

10. The method of claim 1, further comprising, prior to the self-attention process, prepending a learnable classification tag to a set of input tags.

11. The method according to claim 1, wherein: The self-attention process consists of performing multi-head self-attention on a collection of input tokens.

12. The method according to claim 11, wherein: Performing multi-head self-attention involves: performing a first normalization on the set of input tokens to generate a first normalized set; Perform a multi-head self-attention operation on the first normalized set; Performing a second normalization on the output of the multi-head self-attention operation to generate a second normalization; and Multilayer perceptron processing is performed on the second normalization to create the final image representation.

13. The method according to claim 12, wherein: The second normalization is performed on the output of the residual-connected multi-headed self-attention operation added to the set of input tokens.

14. The method according to claim 13, wherein: The final image representation is obtained by adding the second normalized input to the output of the multilayer perceptron processing.

15. The method of claim 1, further comprising pre-training the transformer on a selected image dataset.

16. The method according to claim 1, wherein: The image tile size is selected based on an average resolution across the native resolution image and the one or more resized variants.

17. An image processing system, comprising: a memory configured to store an image; as well as One or more processors operably coupled to the memory, the one or more processors configured to: encoding, for a set of images including a native resolution image and one or more resized variants of the native resolution image, a corresponding spatial embedding of each image block associated with a corresponding region of a given one of the native resolution image or the one or more resized variants to form a set of spatially encoded image blocks; applying the set of scale embeddings to the set of spatially encoded image patches to capture scale information associated with a given one of the native resolution image and the one or more resized variants to form a set of input tags; as well as The final image representation is created via a self-attention process over a set of input tokens.

18. The image processing system according to claim 17, wherein: The one or more processors are further configured to store at least one of the final image representation, the native resolution image, or the one or more resized variants in the memory.

19. The image processing system according to claim 17, wherein: The final image representation corresponds to the predicted quality score of the native resolution image.

20. The image processing system according to claim 17, wherein: The one or more resizing variants form a set of aspect ratio preserving resizing variants.

21. The image processing system according to claim 17, wherein: The one or more processors are further configured to prepend a learnable classification tag to the set of input tags prior to self-attention.

22. The image processing system according to claim 17, wherein: Each resized variant is formed such that the aspect ratio of the variant is resized according to the longer side of the native resolution image.

Images