Multiscale Transformer for Image Analysis
By using a multi-scale transformer and a self-attention mechanism, the limitation of fixed input size in image analysis is overcome, enabling efficient analysis of images with different aspect ratios and resolutions, and improving the accuracy of image quality assessment and classification.
Patent Information
- Application Number
- CN202411172898.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-01
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-07-01
AI Technical Summary
Existing image analysis methods are limited by fixed-size input requirements, which necessitates resizing or cropping images for quality assessment and classification, introducing distortion and low efficiency, and making training on large datasets impractical.
A multi-scale transformer based on image patches is employed to capture fine-grained and coarse-grained information of images through multi-scale representation and self-attention mechanism. Hash-based two-dimensional spatial embedding is used to process images with arbitrary aspect ratios and resolutions, and scale embedding is introduced to distinguish image patches of different scales.
It effectively processes images with different aspect ratios and resolutions, improving the efficiency and accuracy of image analysis. It is suitable for a wide range of image analysis tasks such as IQA, image classification, and multimodal learning.
Smart Images

Figure CN119107305B_ABST
Abstract
Description
[0001] This case is a divisional application of the invention patent application filed on July 1, 2021, with application number 202180100108.6 and invention title "Multi-scale Transformer for Image Analysis". Background Technology
[0002] Image analysis is used in many different types of applications. Some types of image analysis, such as image quality assessment (IQA), quantify the perceived quality of an image. Other types of image analysis can be used to classify aspects of an image, such as objects appearing within it. Such analysis can include analyzing images using convolutional neural networks (CNNs). However, such methods can be limited by the requirement of fixed-size inputs for training models with mini-batches. Resizing or cropping images to fit a given size (with an associated aspect ratio) can affect image synthesis, introduce distortion or other artifacts, and thus alter image quality. Attempts to address such technical problems can introduce other issues, such as the need for large amounts of data storage, the need for resizing and intelligent grouping, or generally make training on large datasets impractical. Summary of the Invention
[0003] The technique employs a patch-based multi-scale transformer, applicable to IQA and other imaging applications. This approach, discussed in detail below, avoids the limitation of a fixed input image size and effectively predicts the quality of native-resolution images. The method transforms the native-resolution image into a multi-scale representation, enabling the transformer's self-attention mechanism to capture information from both fine-grained detailed image patches and coarse-grained global image patches. Self-attention can focus on the entire input sequence, thus effectively capturing image quality at different granularities. A hash-based two-dimensional spatial embedding is used, mapping patch locations to a fixed grid to efficiently handle images with arbitrary aspect ratios and resolutions. Furthermore, since patch locations at each scale are hashed to the same grid, it aligns spatially compact patches across different scales, allowing the transformer model to utilize information across multiple scales. In addition to the spatial embedding, an independent scale embedding is introduced to help the transformer distinguish between patches of different scales in the multi-scale representation. Various embodiments of the disclosed technology can solve technical problems, such as inefficient image analysis due to the need to change the aspect ratio and / or resolution of the input image. The disclosed method also enables better image analysis on a wider range of different images (i.e., not just images that already conform to a specific aspect ratio).
[0004] According to one aspect of the technology, a method for processing an image is provided. The method includes: constructing a multi-scale representation of a native-resolution image via one or more processors, the multi-scale representation including the native-resolution image and a set of resized variants that maintain aspect ratio preserving; encoding, via one or more processors, a corresponding spatial embedding for each image patch associated with a corresponding region of either the native-resolution image or the set of resized variants that maintain aspect ratio preserving, thereby forming a set of spatially encoded image patches; applying, via one or more processors, the set of scale embeddings to the set of spatially encoded image patches to capture scale information associated with the native-resolution image and the set of resized variants that maintain aspect ratio preserving, thereby forming a set of input tokens; and performing self-attention on the set of input tokens via one or more processors according to a transformer encoder module to create a final image representation.
[0005] In one example, the final image represents the predicted quality score corresponding to the native resolution image. Constructing a multi-scale representation may include segmenting the native resolution image and each of the aspect ratio-preserving resized variants into fixed-size image patches, where each image patch represents a different region of either the native resolution image or the aspect ratio-preserving resized variant.
[0006] In another example, encoding the corresponding spatial embedding involves hashing the image patch location for each image patch within the grid of learnable embeddings. In a further example, each aspect-preserving resized variant is formed such that the aspect ratio of the variant is resized according to the longer side of the native resolution image. The multi-scale representation may include a native resolution image with height H, width W, and one or more channels C, and a set of aspect-preserving resized variants is derived using a Gaussian kernel, where each channel represents a color component of the native resolution image.
[0007] In yet another example, to align the set of spatially encoded image patches across scales, the locations of image patches at all scales are mapped to the same grid. In a further example, the method also includes pre-setting learnable classification labels to the set of input labels before self-attention.
[0008] In another example, performing self-attention on a set of input labels includes performing multi-head self-attention on the set of input labels, comprising: performing a first normalization on the set of input labels to generate a first normalized set; performing a multi-head self-attention operation on the first normalized set; performing a second normalization on the output of the multi-head self-attention operation to generate a second normalization; and performing multilayer perceptron processing on the second normalization to create a final image representation. In this case, the second normalization can be performed on the output of the multi-head self-attention operation added to the residual connections of the set of input labels. The final image representation can be obtained by adding the second normalized input to the output of the multilayer perceptron processing.
[0009] In yet another example, the method also includes a transformer for a transformer encoder module pre-trained on a selected image dataset. The size of the image patch can be selected based on the average resolution of a set of resized variants across native resolution images and maintaining aspect ratio. Here, the size of the image patch can be further based on the size of the grid to which the image patch is mapped.
[0010] According to another aspect of the technology, an image processing system includes a memory configured to store an image and one or more processors operatively coupled to the memory. The one or more processors are configured to: construct a multi-scale representation of a native-resolution image, the multi-scale representation including the native-resolution image and a set of aspect ratio-preserving resized variants; encode a corresponding spatial embedding for each image patch associated with a corresponding region of either the native-resolution image or the set of aspect ratio-preserving resized variants, thereby forming a set of spatially encoded image patches; apply the set of scale embeddings to the set of spatially encoded image patches to capture scale information associated with the native-resolution image and the set of aspect ratio-preserving resized variants, thereby forming a set of input tags; and perform self-attention on the set of input tags using a transformer encoder module to create a final image representation.
[0011] In one example, the processor is also configured to store in memory at least one of a set of final image representations, native resolution images, or aspect-preserving resized variants. The final image representation may correspond to a predicted quality score of the native resolution image. Constructing the multi-scale representation may include segmenting the native resolution image and each of each aspect-preserving resized variant into fixed-size image patches, where each image patch represents a different region of one of the native resolution image or aspect-preserving resized variants. Encoding the corresponding spatial embedding includes hashing the image patch location of each image patch within a grid of learnable embeddings. One or more processors may also be configured to pre-feed learnable classification labels to a set of input labels prior to self-attention.
[0012] In yet another example, performing self-attention on a set of input labels includes performing multi-head self-attention on the set of input labels, comprising: performing a first normalization on the set of input labels to generate a first normalized set; performing a multi-head self-attention operation on the first normalized set; performing a second normalization on the output of the multi-head self-attention operation to generate a second normalization; and performing multilayer perceptron processing on the second normalization to create a final image representation.
[0013] According to another aspect of the technology, a method for processing an image is provided, the method comprising: encoding, by one or more processors, a set of images including a native resolution image and one or more resized variants of the native resolution image, and a corresponding spatial embedding of each image patch associated with a given region of the native resolution image or one or more resized variants, to form a set of spatially encoded image patches; applying, by one or more processors, a set of scale embeddings to the set of spatially encoded image patches to capture scale information associated with the native resolution image and one or more resized variants, to form a set of input labels; and creating a final image representation by one or more processors via a self-attention process on the set of input labels.
[0014] According to another aspect of the technology, an image processing system includes: a memory configured to store images; and one or more processors operatively coupled to the memory, the one or more processors being configured to: encode a set of images including a native resolution image and one or more resized variants of the native resolution image, and a corresponding spatial embedding of each image patch associated with a given region of the native resolution image or one or more resized variants, to form a set of spatially encoded image patches; apply a set of scale embeddings to the set of spatially encoded image patches to capture scale information associated with a given one of the native resolution image and one or more resized variants, to form a set of input labels; and create a final image representation via a self-attention process on the set of input labels. Attached Figure Description
[0015] Figure 1 Examples of using cropped or resized images are shown in a CNN-based model.
[0016] Figure 2 Examples of multiscale image representations based on various aspects of this technique are shown.
[0017] Figure 3 An overview of example models of multiscale transformers for image quality assessment based on various aspects of this technique is shown.
[0018] Figures 4A-4C A multi-resolution image representation with maintained aspect ratio is shown according to various aspects of the present technology.
[0019] Figure 4D-Figure 4G The model formation stages according to various aspects of this technology are shown.
[0020] Figure 5A An example converter encoder according to various aspects of this technology is shown.
[0021] Figure 5B The characteristics of multi-head self-attention according to various aspects of this technology are illustrated in the diagram.
[0022] Figure 6 Examples of attention to the original and resized images of a multi-scale composition are shown according to various aspects of this technique.
[0023] Figures 7A-7C A visualization of HSE cosine similarity learned according to various aspects of this technique is shown.
[0024] Figures 8A-8D A visualization of each image in the multi-scale representation according to various aspects of this technique is shown.
[0025] Figures 9A-9B A system for various aspects of this technology is shown.
[0026] Figure 10 Methods according to various aspects of this technology are shown. Detailed Implementation
[0027] Overview
[0028] The image patch-based multi-scale transformer is configured to construct a multi-scale image representation as input. This representation comprises a set of native resolution images and resized variants that preserve aspect ratio (ARP). As discussed further below, each image is segmented into fixed-size image patches, which are embedded by an image patch encoding module. To capture the two-dimensional structure of the image and handle images with different aspect ratios, spatial embeddings are encoded by hashing the patch locations within a learnable embedding grid. Scale embeddings are introduced to obtain scale information. The transformer encoder receives input labels and performs multi-head self-attention. The output representing the entire multi-scale input can be predicted by adding classification labels to the sequence, and the corresponding transformer output is used as the final representation.
[0029] Although the specification is primarily described with reference to Image Quality Analysis (IQA), it should be understood that the disclosed techniques are not limited to such applications but have broad applicability. For example, applications that can utilize this technique include IQA, image classification, image representation learning where tasks require learning representations of images, and multimodal learning (both text and image). In an implementation that trains a patch-based multiscale transformer to predict quality scores, the predicted quality scores can be further used for applications such as image ranking, image recommendation, and filtering of poorly performing images. The technique is also applicable to subtasks in IQA, such as estimating certain technical properties of images (e.g., brightness, contrast, compression ratio). Image classification using the methods discussed herein can involve tasks such as general object classification, as well as classification in biology or medicine where image size should not be resized.
[0030] By being able to process images with their original aspect ratio, the transformer described in this paper can achieve superior results on a wide range of image analysis tasks when compared with existing systems (e.g., as measured by metrics related to the accuracy of the transformer output).
[0031] Figure 1 An example method 100 is shown, in which a CNN-based model is used for image quality assessment. As shown, in the CNN-based model, the full-size input image 102 needs to be resized or cropped to a fixed shape 104 for batch training. However, such preprocessing can change the image aspect ratio and / or composition, which can adversely affect image quality. Conversely, as... Figure 2 As shown in view 200, the multi-scale transformer method based on image patches can process a full-size image 202 into a detailed view, and from resized images 2041,...,204 k The set is used to extract multi-scale features.
[0032] Overall Architecture
[0033] Figure 3 A diagram 300 provides a model overview of a multi-scale transformer according to various aspects of the present technology, which can be used, for example, for image quality assessment. As shown in 302, a multi-scale image representation is constructed as input to the transformer encoder. This representation includes a native resolution image (304) and its ARP-resized variants (3061,...,306). k In this example, each image 304 and 306 is segmented into fixed-size image patches 308, which are embedded by the image patch encoding module 310 to form a multi-scale image patch embedding set 312. To capture the 2D structure of the images and process images with different aspect ratios, the image patch positions (i,j) are hashed to (t) within a learnable embedding grid. i ,t jSpatial embeddings are encoded as shown in the G x G array 314, and the resulting set of embeddings 316 is shown above the multi-scale image patch embedding set 312. A set of scale embeddings 318 is introduced to capture scale information. The transformer encoder module 320 acquires the resulting set of input tokens and performs multi-head self-attention. To obtain the output of the image (e.g., a prediction of image quality or image classification), a classification token [CLS] 322 can be applied to the sequence to represent the entire multi-scale input. The corresponding transformer output can be used as the final representation associated with the native resolution image, such as a quality score or classification (not shown). Various aspects of the model will be described in detail below.
[0034] More specifically, a multi-scale representation of the input image is created, which includes the original resolution image and its ARP-resized variant. For example... Figure 4A As shown in Example 400, although there may be three, four or more variations of the resizing, there are two variations of the resizing. Variations of the resizing can be formed such that the aspect ratio is adjusted according to the longer side of the original image (i.e., the longer side is fixed to L_i as shown).
[0035] Images of different scales are divided into fixed-size image blocks and fed into the model. Figure 4B View 410 is shown, in which image patches 412 are displayed as different boxes for each image. Because the image patches come from images of different resolutions, the system effectively encodes multi-aspect ratio, multi-scale input into a sequence of labeled blocks. Figure 3 The image block coding module (frames 312, 316, and 318) captures pixel, spatial, and scale information. Figure 3 (310) encodes image patches extracted from the multi-scale representation. A hash-based spatial embedding module encodes the 2D spatial location of each image patch. Furthermore, learnable scale embeddings encode different scales. After encoding the multi-scale input into a labeled sequence, the system can pre-load additional learnable "classification labels" [CLS] (see [link]). Figure 3 (322). In this case, the CLS-labeled state at the transformer encoder output is used as the final image representation. Fully connected layers can be added to predict image quality scores. The labels are fixed-size vectors. The final input labels include scale, spatial information, and image patch information.
[0036] Multi-scale image patch embedding
[0037] Image quality can be affected by both local details and global composition. To capture both global and local information simultaneously, this technique uses a multi-scale representation to model the input image. Image patches of different scales allow the transformer to aggregate information across multiple scales and spatial locations. In one example, the multi-scale input consists of a sequence of images: a full-size image with height H, width W, and channels C, and an ARP-resized full-size image using a Gaussian kernel. Each channel represents a color component of the image. For example, when the input image is in RGB format, it has 3 channels. A grayscale image has only 1 channel, and an RGBA image has 4 channels (RGB+alpha). For an RGB image, each pixel is represented by three [0, 255] values of pixel intensity, each representing the R, G, and B channel values, respectively. Therefore, an RGB image with height H and width W can be represented by a shape matrix [H, W, 3].
[0038] Here, the resized image has a height h. k Width w k And channels C, where k = 1,...,K, where K is the total number of resized variants for each input. To align the resized images for a consistent global view, the longer side length is fixed at L for each resized variant. k Adjust the size factor α k It can be defined as:
[0039] α k = L k / max (H,W), where h k = α k H, w k = α k W (1)
[0040] Depending on the scenario, there may be no restrictions on how to resize the image, as long as the length remains consistent during training and testing.
[0041] Figure 4C View 420 is shown, illustrating the "cropping" (image patch) 422 of the original resolution image embedding, and the croppings 4241 and 4242 of the resized image (K=2). These form the pixel embedding portion of the image patch model. Once preprocessing is complete, the cropping is input to the image patch encoding module (…). Figure 3 (310 in the middle).
[0042] In multi-scale representation, square image patches of size P are extracted from each image (cropping). For example, the image patch size P can be 16x16, 32x32, 64x64, or larger or smaller. For images whose width or height is not a multiple of P, the image can be padded with zeros accordingly. Each image patch is encoded as a D-dimensional embedding by an image patch encoding module. Here, D is the latent label size used in the transformer. While image patches can be encoded using linear projection, in one case, a 5-layer ResNet-type convolutional neural network with fully connected layers of size D can be used as the image patch encoder module to learn a better representation of the input image patches. When pre-trained on ImageNet Large Scale Visual Recognition Challenge (ILSVRC) such as ILSVRC-2012 ImageNet, encoding image patches using several convolutional layers (e.g., up to 5 layers) may perform better than linear projection. Since the image patch encoding module is lightweight and shared across all input image patches of small size P, it adds only a small number of parameters. For example, the table below shows examples of the total number of parameters for models with different image patch encoding modules using the AVA dataset. Here, for a 32x32 image patch size, 2 layers of convolution add 1 million (1M) parameters compared to linear projection, while 5 layers of ResNet add 5 million (5M) parameters. For example, Spearman rank ordered correlation (SRCC), Pearson linear correlation (PLCC), and 2 layers of convolution can use the roots of ResNet (7x7 convolutions followed by 3x3 convolutions). For 5 layers of ResNet, residual blocks are stacked on top of 2 (simple) convolutions.
[0043] #parameter SRCC PLCC Linear projection 22M 0.703 0.716 2-layer convolution 23M 0.717 0.731 5-layer ResNet 27M 0.726 0.738
[0044] Table 0: Comparison of different image patch coding modules on the AVA dataset
[0045] The image patch embedding output sequences from the image patch coding module are concatenated to form a multi-scale embedding sequence of the input image. The number of image patches for the original image and the resized image can be calculated as N = HW = P, respectively. 2 and n k =h k w k / P 2 Therefore, the number of image patches in each image may be different.
[0046] Since each input image has a different resolution and aspect ratio, H and W are different for each input, therefore N and nk The difference lies in the input length. To obtain a fixed-length input during training, the encoded image patch markers can be zero-padding or clipped to the same length. An input mask can be attached to indicate the valid input, which will be used in the transformer to perform masked self-attention. It's important to note that padding does not change the input because in multi-head attention, the padding markers are ignored by masking.
[0047] As mentioned above, for each resizing variant, the length of the longer side is fixed at L. k .therefore, And the system can safely fill up to m k For native resolution images, the system can simply pad or cut the sequence to a fixed length l. Because the sequence length can be arbitrary, padding is not necessary during single-input evaluation.
[0048] Hash-based 2D spatial embedding
[0049] Spatial location embedding is beneficial for injecting awareness of 2D image structure into 1D sequence inputs. Fixed-length location embedding specifies an embedding for each input location; however, this fails for variable input resolutions with varying numbers of image patches, as each image patch in the sequence may originate from an arbitrary location in the image. Furthermore, independently localized location embedding models cannot align spatially compact image patches from different scales. Therefore, aspects of this technique are configured to meet the following requirements: First, to efficiently encode spatial information of image patches at different aspect ratios and input resolutions. Second, spatially compact image patches at different scales should have tight spatial embeddings. Third, the embedding should be efficient and easy to implement, and non-intrusive to the transformer attention. Here, non-intrusiveness, compared to relative location embedding methods that require changes to the transformer attention implementation, means that the system does not need to modify the transformer attention implementation.
[0050] In view of the above, a hash-based 2D spatial embedding (HSE) is provided, where the image patch located at row i, column j is hashed to the corresponding element in a G x G grid (see [link to HSE]). Figure 3 (314 in the middle). Figure 4D View 430 in the diagram illustrates how the input crop of the image patch is hashed or otherwise mapped to corresponding elements in a G x G grid. For example, a lookup table can be used to implement hashing.
[0051] Each element in the grid is a D-dimensional embedding. The HSE is composed of a learnable matrix. Definition. Assume the input resolution is H x W. The input image will be divided into... Image patch. For an image patch at position (i,j), its spatial embedding is determined by position (t) in T. i ,t jThe element definition at () is as follows:
[0052]
[0053] like Figure 3 As shown, embedding D-dimensional space Add to the image block embedding element by element. Figure 4E View 440 shows the spatial embedding of the cropped image block pixel embeddings 442, 4441 and 4442 applied for each resolution.
[0054] For fast lookups, the system can simply use (t) i t j Round to the nearest integer. HSE does not require any changes to the transformer attention module. Furthermore, t i and t j The computation and lookup are both lightweight and easy to implement, requiring no unnecessary computing resources.
[0055] To align image patches across scales, the positions of image patches at all scales are mapped to the same grid T. Therefore, image patches that are close together in the image but at different scales will be spatially tightly embedded in T, as i and H, and j and W, change proportionally with the resizing factor α. This achieves spatial alignment across different images represented at multiple scales.
[0056] There is a trade-off between expressiveness and trainability when choosing the hash grid size G. A smaller G (e.g., 4 or less) may result in excessive collisions between image patches, making it difficult for the model to distinguish spatially compact patches. A larger G (e.g., 100 or more) may consume significant memory resources and may require training at more diverse resolutions. A larger G means a sparser hash, thus requiring more diverse resolutions for training, otherwise some locations may not have enough data to learn good representations. For learnable T, a suitable rule of thumb is to let the grid size multiplied by the number of image patches P roughly equal the average resolution, so G x G x P x P = H x W. For different datasets with an average resolution of approximately 450x500 and an image patch size of 32, the grid size might be between 10 and 15. In one scenario, based on an IQA setting where coarse location information is sufficient, it was found that once G is large enough, changing G only results in small performance differences. Therefore, in one example, G could be between 5 and 15, such as 10, although larger or smaller G values could be used.
[0057] Scale embedding
[0058] Since the same hash matrix can be reused for all images, HSE does not distinguish between image patches of different scales. Therefore, Additional Scale Embedding (SCE) is introduced to help the model effectively distinguish information at different scales and better utilize cross-scale information. In other words, SCE labels which input scale an image patch comes from in a multi-scale representation. Figure 4F View 450 shows the scale embeddings 452, 4541, and 4542 applied to the spatial embeddings and 442, 4441, and 4442, as well as the cropped image patch pixel embeddings 442, 4441, and 4442 for each resolution. As mentioned above, in order to obtain a fixed-length input during training, the encoded image patch tags can be zero-padded to the same length. This is in Figure 4G As shown in view 460, zero-filled blocks 462, 4641, and 4642 are used.
[0059] SCE is defined as a learnable scale embedding of an input image with a variant that has been resized to k-scale. After spatial embedding, the first element It is added element-wise to all D-dimensional image patch embeddings of the native resolution image. It is also added element-wise to the embedding of all image blocks in the image resized by scale k.
[0060] Pre-training and fine-tuning
[0061] In one example, the transformer model is pre-trained on a large dataset (e.g., a dataset with millions or more images, such as ImageNet) and fine-tuned on a downstream task. During pre-training, the system can keep random cropping as augmentation to generate images of different sizes. However, the system might skip resizing and initialize the model for inputs with different resolutions and aspect ratios instead of performing square resizing. Augments such as RandAugment and mixup can be used during pre-training. For example, for an augmentation operation pool, RandAugment randomly selects N from the pool and applies it sequentially by magnitude M (N and M are configurable). Mixup is a data augmentation technique that generates a weighted combination of random image pairs from the training data. Other suitable pre-training datasets include ImageNet21k, JFT-300M, and OpenImage.
[0062] When fine-tuning for the IQA task, the system does not need to adjust or crop the input image to maintain image composition and aspect ratio, thus allowing the model to start with an unmodified input image. In one scenario, the only augmentation used in fine-tuning is random horizontal flipping. Here, for each training image, the system can randomly flip it horizontally with probability p, where p can be 0.5 (or more or less). For evaluation, this method can be applied directly to the original image without aggregating multiple augmentations (e.g., multi-cropping sampling).
[0063] When fine-tuning an IQA dataset (which may include images labeled with corresponding IQA measurements / scores), regression losses (such as L1 loss for a single mean opinion score (MOS) and Earth Mover Distance (EMD) loss) can be used to predict the quality score distribution:
[0064]
[0065] Where p is the normalized score distribution and CDF p (m) is the cumulative distribution function EMD loss penalizes the distribution distance between ratings and predictions. Here, r is a hyperparameter used to select the r-norm (r-normalized) distribution distance. For example, r can be set to 2 to penalize the distance between CDFs using the Euclidean distance.
[0066] When the system is trained to perform other image analysis tasks, other training datasets and loss functions (or objective functions) can be used. For example, for a classification task, a classification dataset including multiple image inputs and corresponding labels representing specific categories of the images can be used to train the transformer. A suitable loss function, such as cross-entropy loss, can be utilized to train the transformer in this scenario. It will be understood that, as previously stated, the transformer can be used for other image analysis tasks, and any suitable dataset and / or loss function can be used to train the transformer to perform image analysis tasks.
[0067] Converter encoder
[0068] Figure 5A An example converter encoder 500 is shown. As illustrated, an embedded multi-scale representation 502 is input to a converter encoder block 504. The converter encoder block 504 includes the following layers: a first-layer normalization 506, a multi-head self-attention (MSA) layer 508, a second-layer normalization 510, and a multilayer perceptron (MLP) layer 512. Residual connections are added between the layers, as indicated by arrows 514 and 516.
[0069] Multi-scale image patches are encoded as Where k = 0...K, it is the scale index, and n is the image patch index in the scale. k = 0 represents a full-size image. K is an adjustable parameter. Hash-based 2D spatial embedding (HSE) and scale embedding (SCE) are added to the image patch embedding to form a multi-scale representation input. Learnable [class] labeled embeddings can be pre-set in the embedded label sequence (x class In ), the converter encoder can be formulated as:
[0070]
[0071] z0 = [x class E p +E HSE +E SCE (5)
[0072] z′ q =MSA(LN(z) q -1))+z q-1 , q=1…L (6)
[0073] z q =MLP(LN(z′) q ))+z′ q , q=1…L (7)
[0074]
[0075] E p It is image patch embedding. E HSE and E SCE These are spatial embedding and scale embedding, respectively. l is the number of image patches at the original resolution. m1...m K This represents the number of image patches in the resized variant. z0 is the input to the transformer encoder. g is the output of each converter layer, and L is the total number of converter layers.
[0076] Using a larger value for l during fine-tuning can improve model performance. Since high-resolution images have more image patches than low-resolution images, when l is too small, some larger images may be truncated, thus degrading model performance. For example, l can be chosen as 128, 256, or 512, or more or less.
[0077] Combination Figure 5B Example 520 discusses self-attention and its multi-head version, multi-head self-attention (MSA). Assume the input sequence is... Let vectors Q, K, and V represent the query (Q), key (K), and value (V), respectively. Q, K, and V are represented using learnable matrices. The input sequence is projected to generate D. h These are the inner dimensions of Q, K, and V. Then attention weights are used. Calculate the weighted sum on V, where the attention weights are the pairwise similarities between Q and K.
[0078] Q = zU q K = zU k V = zU v (9)
[0079]
[0080] SA(z)=AV (11)
[0081] MSA is an extension of self-attention, where *s* self-attention operations (heads) are performed in parallel. The outputs of all heads are concatenated and then processed using a learnable matrix. Projected onto the final output. D h It can be set to D / s to keep the computation and number of parameters per s constant. This will result in:
[0082] MSA(z) = [SA1(z); ...; SA s (z)]U M (12)
[0083] In batch training, input masks are used to indicate valid input and ignore padding markers. For example... Figure 5B As shown in block 526, a mask is added to the attention weights before the softmax function is executed in block 528. Before the softmax step in Equation 10, the attention weights at invalid locations are close to zero by setting the corresponding elements to -inf (negative infinity). In one example, considering the mixing of attention on the labels through all layers, the attention weights are averaged over all heads and then recursively multiplied. Therefore, Figure 5B Represents formulas (10) and (11). The MatMul block 522 represents matrix multiplication (QK in formula 10). T The scale at block 524 is represented in (Equation 10) according to The scaling. The masking operation at block 526 sets the elements to be ignored to -inf (see Equation 14 below). Softmax block 528 will... The output is normalized to a probability distribution. The final MatMul block 530 is the AV multiplication in Equation 11.
[0084] Attention masks can be constructed as Where: if attention
[0085]
[0086] Then, the self-attention weight matrix of the mask can be calculated as follows:
[0087]
[0088] Example Implementation
[0089] In one example, the multi-scale representation is constructed as a native resolution image and resized using two ARP scales, L1 = 224 and L2. Therefore, the number of scales is K = 3. The image patch size is P = 32. The transformer input label has a dimension of D = 384, which is also the dimension of the pixel image patch embedding, HSE, and SCE. The HSE grid size is set to G = 10. A transformer with lightweight parameters (e.g., 384 hidden size, 14 layers, 1152 MLP size, and 6 heads) is employed to make the model size comparable to ResNet-50. The final model has approximately 27 million parameters.
[0090] In this example, Adam is used, with β1 = 0.9; β2 = 0.999, a batch size of 4096, weight decay of 0.1, and a cosine learning rate decaying from 0.001. The MST-IQA model was trained 300 times on ImageNet. During training, the maximum number of image patches was set from 1 to 512 for full-size images. For fine-tuning, SGD, along with momentum and cosine learning rates decaying from 0.0002, 0.002, 0.0001, and 0.12, was used for 10, 30, 30, and 20 times on PaQ-2-PiQ, KonIQ-10k, SPAQ, and AVA, respectively. For AVA, the batch size was set to 512; for KonIQ-10k, the batch size was set to 96; and for the others, the batch size was set to 128. For AVA, we used EMD loss with r = 2. For the other datasets, L1 loss was used.
[0091] The model can be trained on one or more Tensor Processing Units (TPUs). All results are averaged over 10 runs. Report SRCC, PLCC, and the corresponding standard deviation (std). Table 1 below shows the results for the PaQ-2-PiQ full-size validation and test sets. Figure 6View 600 graphically illustrates the attention process for both the original and resized images. The last row of the tables in View 600 (full size, 384, 224) shows the test set results from Table 1. The image on the right is a visualization of the attention process from the output labels to multi-scale representations (original resolution images and two ARP resized variants). The images in this paper were resized to fit the grid, and the model input was three different resolutions. The model focuses on details in high-resolution images and on the global scale in low-resolution images. This demonstrates that the model learns image quality at different granularities. Table 2 shows the results for the KonIQ-10k dataset. Table 3 shows the results for the SPAQ dataset. Table 4 shows the results for the AVA dataset.
[0092]
[0093] Table 1: Full-size validation and test sets for PaQ-2-PiQ
[0094] SRCC PLCC 0.909 0.923
[0095] Table 2: Results of the KonIQ-10k dataset
[0096] SRCC PLCC 0.917 0.921
[0097] Table 3: Results of the SPAQ dataset
[0098] Classification accuracy Mean square error SRCC PLCC 0.815 0.242 0.726 0.738
[0099] Table 4: Results of the AVA dataset
[0100] Ablation can be used to evaluate the effectiveness of hash-based spatial embedding (HSE) and scale-based embedding (SCE). Table 5 below shows ablation studies of spatial embedding on AVA.
[0101] Spatial Embedding SRCC PLCC w / o embedded 0.704 0.716 Fixed length (no HSE) 0.707 0.722 HSE 0.726 0.738
[0102] Table 5: Ablation Study Results of Spatial Embedding on AVA
[0103] For "fixed-length (without HSE)," the input is treated as a fixed-length sequence, and a learnable embedding is assigned to each location. As shown here, there is a significant difference between adding and not adding spatial embeddings ("without embedding"). Even fixed-length sequences are less satisfactory than HSE. This is for two reasons. First, the input has different aspect ratios. Therefore, each image patch in the sequence can come from a different location in the image. Fixed-location embeddings cannot capture this variation. Second, since each location is modeled independently, there is no cross-scale information, meaning the model cannot locate spatially compact image patches of different scales in multi-scale representations. Furthermore, the fixed-length approach is inflexible because fixed-length spatial embeddings cannot be easily applied to large images with more image patches. In contrast, HSE is meaningful in all cases.
[0104] exist Figure 7A View 700 (G=10), Figure 7B View 710 (G=5) and Figure 7C A visualization of the learned HSE cosine similarity is provided in view 720 (G=15). As these views show, HSEs in spatially close locations are more similar (lighter shading or color, or brighter in the same general location), and it corresponds well to 2D structure. For example, the bottom HSE is the brightest at the bottom. This shows that HSE can effectively capture the 2D structure of an image.
[0105] Table 6 shows that adding SCE can further improve performance compared to not adding SCE. This indicates that SCE helps the model capture scale information independently of spatial information.
[0106] Scale embedding SRCC PLCC w / o embedded 0.717 0.729 With embedded 0.726 0.738
[0107] Table 6: Ablation Study Results of Upper-Scale Embedding in AVA
[0108] Different designs can be used to encode image patches, including linear projection and a small number of convolutional layers. As shown in Table 7, performance can be improved by using a simple convolution-based image patch encoding module. For simple convolutions, the root of ResNet (7x7 convolutions, then 3x3 convolutions) is used. For a 5-layer ResNet, residual blocks are stacked on top of simple convolutions. Adding more convolutional layers may have diminishing returns, and it can be seen that a 5-layer ResNet can provide a satisfactory representation of image patches.
[0109] #parameter SRCC PLCC Linear projection 22M 0.703 0.716 Simple convolution 23M 0.717 0.731 5-layer ResNet 27M 0.726 0.738
[0110] Table 7: Comparison of different image patch coding modules on the AVA dataset
[0111] Table 8 shows the evaluation of the impact of image patch size. Here, ablation was performed on different image patch sizes P. From this set of results, it can be seen that image patches of size P = 32 exhibit the most effective performance.
[0112]
[0113]
[0114] Table 8: Comparison of different image patch sizes on the AVA dataset
[0115] Figure 10 Visualization of attention from output labels to multi-scale representation. The "Input" column displays the input images. "Attn Full" shows the attention for the full-size image. "Attn L=384" and "Attn L=224" show the attention for images resized by ARP. Note that the images here are resized to fit the mesh, and the model input is at 3 different resolutions.
[0116] To understand how the system uses self-attention to integrate information at different scales, a visualization of the average attention weights from the output labels to each image can be provided for the multi-scale representation. Examples of this are shown in [references to be inserted here]. Figures 8A-8D The examples are given in views 800, 810, 820, and 830. In these examples, the attention weights are averaged across all heads and then recursively multiplied, taking into account the mixed attention across the labels in all layers.
[0117] The image patch-based multi-scale transformer based on the features disclosed herein can be implemented using TPUs, CPUs, or other computing architectures. An example computing architecture is as follows: Figure 9A and Figure 9B As shown. In particular, Figure 9A and Figure 9B These are schematic and functional diagrams of an example system 900, which includes multiple computing devices and a database connected via a network. For example, computing device 902 could be a cloud-based server system. Databases 904, 906, and 908 can store, for example, raw full-size images, multi-scale representations, and / or transformer modules (such as transformer encoding modules, image patch encoding modules, etc.), respectively. The server system can access the database via network 910. Client devices may include one or more of a desktop computer 912 and a laptop or tablet PC 914, for example, for providing raw images and / or for viewing output visualizations.
[0118] like Figure 9BAs shown, each of computing devices 902 and 912-914 may include one or more processors, memory, data, and instructions. Memory stores information accessible to one or more processors, including instructions and data (e.g., models) that can be executed or otherwise used by the processor. Memory can be any type capable of storing processor-accessible information, including computing device-readable media. Memory is a non-transitory medium, such as a hard disk drive, memory card, optical disk, solid-state drive, etc. The system may include different combinations of the above, thereby storing different portions of instructions and data on different types of media. Instructions can be a collection of any instructions that are directly executed by the processor (such as machine code) or indirectly executed (such as scripts). For example, instructions may be stored as computing device code on a computing device-readable medium. In this context, the terms "instruction," "module," and "program" are used interchangeably. Instructions may be stored in object code format for direct processor processing or in any other computing device language, including scripts or collections of independent source code modules that are interpreted on demand or pre-compiled.
[0119] The processor can be any conventional processor, such as a commercially available CPU, TPU, graphics processing unit (GPU), etc. Alternatively, each processor can be a dedicated device, such as an ASIC or other hardware-based processor. Although Figure 9B Functionally, a given computing device is shown to have its processor, memory, and other components housed within the same block; however, such devices may actually include multiple processors, computing devices, or memories that may or may not be housed within the same physical enclosure. Similarly, memory may be a hard drive or other storage medium located in an enclosure different from that of the processor, such as in a cloud computing system like server 902. Accordingly, references to processors or computing devices will be understood to include references to a collection of processors, computing devices, or memories that may or may not operate in parallel.
[0120] Input data, such as one or more raw full-size images, can be manipulated by the transformer module to generate one or more multi-scale representations, image quality assessment data, etc. Client devices can then utilize this information in various applications or other programs to perform image ranking, image recommendation, image classification, and / or bad image filtering.
[0121] The computing device may include all components typically used in conjunction with computing devices such as the processor and memory described above, as well as a user interface subsystem for receiving input from a user and presenting information (e.g., text, images, and / or other graphical elements) to the user. The user interface subsystem may include one or more user inputs (e.g., at least one front-facing (user) device facing a camera, a mouse, keyboard, touchscreen, and / or microphone) and one or more display devices (e.g., a monitor with a screen or any other electrical device operable to display information (e.g., text, images, and / or other graphical elements)). Other output devices, such as speakers, may also provide information to the user.
[0122] User-related computing devices (e.g., 912-914) can communicate with backend computing systems (e.g., server 902) via one or more networks (such as network 910). Network 910 and intermediate nodes can include various configurations and protocols, including those such as Bluetooth. TM Short-range communication protocol, Bluetooth LE TM The Internet, World Wide Web, Intranet, Virtual Private Network, Wide Area Network, Local Area Network, Private Network using one or more proprietary communication protocols, Ethernet, WiFi, and HTTP, as well as various combinations thereof. This communication can be facilitated by any device capable of sending and receiving data to and from other computing devices, such as modems and wireless interfaces.
[0123] In one example, computing device 902 may include one or more server computing devices having multiple computing devices, such as a load-balanced server farm or cloud computing system, which exchanges information with different nodes in the network for receiving, processing, and sending data to other computing devices. For example, computing device 902 may include one or more server computing devices capable of communicating with any of computing devices 912-914 via network 910.
[0124] Image quality assessment information or other data derived from the transformer module, the module itself, multi-scale representations, etc., can be shared by the server and one or more client computing devices. Alternatively or additionally, client devices can maintain their own databases, transformer modules, etc.
[0125] Figure 10A method 1000 for processing an image according to various aspects of this technique is illustrated. The method includes: at block 1002, constructing a multi-scale representation of a native resolution image. The multi-scale representation includes the native resolution image and a set of aspect ratio-preserving resized variants. At block 1004, the method includes encoding a corresponding spatial embedding for each image patch associated with a corresponding region of one of the native resolution image or the set of aspect ratio-preserving resized variants, thereby forming a set of spatially encoded image patches. At block 1006, the method includes applying a set of scale embeddings to the set of spatially encoded image patches to capture scale information associated with the native resolution image and the set of aspect ratio-preserving resized variants, thereby forming a set of input tags. And at block 1008, the method includes performing self-attention on the set of input tags according to a transformer encoder module to create a final image representation.
[0126] The image patch-based multiscale transformer method discussed in this paper is advantageous for many reasons and has broad applicability. For example, applications that can be used include IQA, image classification, image representation learning where tasks require learning image representations, and multimodal learning (text and images). The predicted quality scores can be further used in applications such as image ranking, image recommendation, and filtering of poorly formatted images. This technique is also applicable to subtasks in IQA, such as estimating some technical properties of images (e.g., brightness, contrast, compression ratio). Image classification using the method discussed in this paper can involve tasks such as general object classification, as well as classification in biology or medicine, where images should not be resized. The system can represent text and images as tags and perform multimodal learning.
[0127] Although the techniques described herein have been referenced to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the technology. Therefore, it should be understood that many modifications can be made to the illustrative embodiments, and other arrangements can be designed, without departing from the spirit and scope of the technology as defined by the appended claims.
Claims
1. A method for processing an image, the method comprising: A spatially encoded set of image patches is formed by encoding a set of images, including a native resolution image and one or more resized variants of the native resolution image, by one or more processors, and the corresponding spatial embedding of each image patch associated with a given region of the native resolution image or one or more resized variants. The set of scale embeddings is applied to a set of spatially encoded image patches by one or more processors to capture scale information associated with a given one of the native resolution image and one or more resized variants to form a set of input tags; as well as The final image representation is created by one or more processors via a self-attention process on a set of input labels.
2. The method according to claim 1, wherein, The final image represents the predicted quality score corresponding to the native resolution image.
3. The method of claim 1, further comprising constructing a multi-scale representation of the native resolution image, the multi-scale representation comprising the native resolution image and the one or more resized variants.
4. The method according to claim 3, wherein, The one or more resizing variants form a set of resizing variants that maintain aspect ratio.
5. The method according to claim 3, wherein, Constructing a multi-scale representation involves segmenting the native resolution image and each resized variant into fixed-size image blocks, where each image block represents a region of either the native resolution image or a resized variant.
6. The method according to claim 3, wherein, The multi-scale representation includes a native resolution image with height H, width W, and one or more channels C, and one or more resized variants are derived using a Gaussian kernel, where each channel represents a color component of the native resolution image.
7. The method according to claim 1, wherein, Encoding the corresponding spatial embedding involves hashing the image patch position of each image patch within the grid of learnable embeddings.
8. The method according to claim 1, wherein, Each resized variant is formed such that the aspect ratio of the variant is resized according to the longer side of the native resolution image.
9. The method of claim 1, further comprising aligning a set of spatially encoded image patches across scales by mapping the locations of image patches at all scales to a grid.
10. The method of claim 1, further comprising, prior to the self-attention process, pre-setting learnable classification tags to a set of input tags.
11. The method according to claim 1, wherein, The self-attention process involves performing multi-head self-attention on a set of input labels.
12. The method according to claim 11, wherein, Performing multi-head self-attention includes: Perform a first normalization on the set of input labels to generate a first normalized set; Perform a multi-head self-attention operation on the first normalized set; Perform a second normalization on the output of the multi-head self-attention operation to generate a second normalized output; and Multilayer perceptron processing is performed on the second normalization to create the final image representation.
13. The method according to claim 12, wherein, The second normalization is performed on the output of the multi-head self-attention operation of the residual connections added to the set of input labels.
14. The method according to claim 13, wherein, The final image representation is obtained by adding the second normalized input to the output of the multilayer perceptron processing.
15. The method of claim 1, further comprising pre-training a transformer on a selected image dataset.
16. The method according to claim 1, wherein, The image patch size is selected based on the average resolution across the native resolution image and the one or more resized variants.
17. An image processing system, comprising: The memory is configured to store images; as well as One or more processors, operatively coupled to the memory, said one or more processors being configured to: Encode the spatial embedding of each image patch associated with a given region of the native resolution image or one or more resized variants of the native resolution image to form a spatially encoded set of image patches. The set of scale embeddings is applied to a set of spatially encoded image patches to capture scale information associated with a given one of the native resolution image and one or more resized variants to form a set of input tags; as well as The final image representation is created through a self-attention process on a set of input labels.
18. The image processing system according to claim 17, wherein, The one or more processors are also configured to store in memory at least one of the final image representation, the native resolution image, or a resized variant of the one or more processors.
19. The image processing system according to claim 17, wherein, The final image represents the predicted quality score corresponding to the native resolution image.
20. The image processing system according to claim 17, wherein, The one or more resizing variants form a set of resizing variants that maintain aspect ratio.
21. The image processing system according to claim 17, wherein, The one or more processors are also configured to pre-feed learnable classification labels to the set of input labels prior to self-attention.
22. The image processing system according to claim 17, wherein, Each resized variant is formed such that the aspect ratio of the variant is resized according to the longer side of the native resolution image.
Citation Information
Patent Citations
Hyper-spectral and panchromatic image fusion method for extracting multi-scale spatial-spectral characteristics based on AE
CN112634137A
Header-based processing of images compressed using multi-scale transforms
EP1329847A1
