Image processing using self-attention

Abstract

An image processing device for identifying one or more features of an input image, the device comprising a processor configured to: receive the input image, the input image extending along a first axis and a second axis; form a series of attribute maps based on the received input image, each attribute map representing an intensity of a respective attribute at a plurality of locations in the image; perform a first correlation operation by identifying regions associated with patterns of a plurality of attribute maps in the series of attribute maps, and form a first output in dependence on the operation; perform a second correlation operation to identify a combination of (i) attributes and (ii) portions of the image having a common location along the first axis, wherein the combination is correlated across a plurality of locations along the second axis, and form a second output in dependence on the operation; and form a representation of the one or more features of the input image in dependence on at least the first output and the second output.

Description

TECHNICAL FIELD

[0001] The present invention relates to image processing, for example for computer vision purposes. BACKGROUND

[0002] It is known to use deep neural networks such as convolutional neural networks (CNNs) for image analysis. In a CNN, an image is processed successively through multiple convolution and non-linear (e.g. rectified linear unit, ReLU) layers to extract features. These features are abstractions of the image data. The features themselves can be processed by further convolution and non-linear layers to convert the features to a further level of abstraction. SUMMARY

[0003] Image features (including features of videos, e.g. video frames) can be described as tensors. A tensor can be thought of as a multi-dimensional matrix. The dimensions of a tensor are referred to as the rank, denoted D. A 0D tensor corresponds to a single number or scalar, a ID tensor corresponds to a vector, a 2D tensor corresponds to a matrix, a 3D tensor corresponds to a 3D array, and so on. A tensor can be thought of as an abstract representation of some data input. Tensors can be used to represent complex data structures such as images and videos in computer vision, text corpora in natural language processing, or gene expression in bioinformatics.

[0004] For illustration purposes, Figure 1 An example of a CNN designed for image classification is shown schematically. The output of each layer is a 3D tensor, which is then input to the next layer, and so on until the final fully connected layer which classifies using the abstract features extracted from the image.

[0005] An ideal property of a CNN is to extract meaningful task-specific features to solve a particular problem, for example, high-level vision problems (such as image classification) or low-level vision problems (such as image inpainting or image-to-attribute mapping). Applications of inpainting or image-to-attribute mapping include forming an image with high perceptual quality (e.g. an image in RGB format) from a source image (e.g. an image in RAW format) captured by a camera sensor or from an image that is somehow corrupted (e.g. due to a missing part of the image). Capturing better features in the source image can have a huge impact on the performance of the CNN.

[0006] Images often exhibit a high degree of self-similarity. For example, an image can include multiple faces. By exploiting this self-similarity, even pixels in the image that are not adjacent to each other (i.e. non-local or distant pixels) can support each other to enrich the features extracted and encoded in the tensor describing the image.

[0007] In the field of computer vision, some traditional image processing operations make use of self-similarity information. A clear example is the famous BM3D image denoising technique, which draws similarities between pairs of blocks of the input image.

[0008] However, the state-of-the-art technique in the field of computer vision and image processing is the convolutional neural network, which generally outperforms traditional methods in various tasks such as demosaicking, denoising, color enhancement. However, in some implementations where computational power is limited, a drawback of these models can be that they only process each input point as a function of its neighboring regions, without considering long-range dependencies in the input.

[0009] Recently, Wang et al. (X Wang, R Girshick, A Gupta, and K He (2018), “Non-local neural networks,” IEEE Conference on Computer Vision and Pattern Recognition

[0010] There is a need for an improved method of performing image processing that takes into account similarities within an image.

[0011] According to an aspect, there is provided an image processing device for identifying one or more features of an input image, the device comprising a processor configured to: receive the input image, the input image extending along a first axis and a second axis; form a series of attribute maps based on the received input image, each attribute map representing an intensity of a respective attribute at a plurality of locations in the image; perform a first correlation operation by identifying regions that are relevant to a pattern of a plurality of attribute maps in the series of attribute maps, and form a first output in dependence on the operation; perform a second correlation operation to identify a combination of (i) attributes and (ii) portions of the image having a common location along the first axis, wherein the combination is correlated across a plurality of locations along the second axis, and form a second output in dependence on the operation; and form a representation of the one or more features of the input image in dependence on at least the first output and the second output.

[0012] The processor can be configured to: perform a third correlation operation to identify a combination of (i) attributes and (ii) portions of the image having a common location along the second axis, wherein the combination is correlated across a plurality of locations along the first axis, and form a third output in dependence on the operation; wherein forming the representation of the one or more features of the input image is further dependent on the third output. By performing self-attention in an additional dimension, further information about the one or more features of the image can be obtained.

[0013] The image can be a still image or a part (e.g. a frame) of a video.

[0014] One of the first and second axes can be an image horizontal axis X. The other of the first and second axes can be an image vertical axis Y. The attributes can form a set C and the image and the attribute maps together can form a tensor of dimensions C, X and Y. This provides a convenient way of analysing the image data.

[0015] The output of the first correlation operation can be a similarity matrix of dimensions X, Y; the output of the second correlation operation can be a similarity matrix of dimensions C and one of X and Y. This provides a convenient way of analysing the image.

[0016] The attributes can include one or more of: a presence rate of a particular hue, a luminosity, a local contrast and a determined representation of a local likelihood of a particular feature. The feature can be a face.

[0017] The processor can be operable to perform a feature recognition operation on the input image to form a map comprising local likelihood estimates of a particular feature at a plurality of locations in the input image. This map can constitute one of the attribute maps. The apparatus can perform feature recognition on the input image to identify features therein and can estimate a local presence rate of such features in response to such a feature recognition process. This can allow feature similarity at spaced locations in the image to be identified.

[0018] The processor can be operable to train a convolutional neural network from the representation. The trained network can then be used for image processing.

[0019] According to a second aspect, there is provided a method of identifying one or more features of an input image, the method comprising: receiving the input image, the input image extending along a first axis and a second axis; forming a series of attribute maps based on the received input image, each attribute map representing an intensity of a respective attribute at a plurality of locations in the image; performing a first correlation operation by identifying regions associated with a pattern of a plurality of the attribute maps in the series of attribute maps and forming a first output in dependence on the operation; performing a second correlation operation to identify a combination of (i) attributes and (ii) portions of the image having a common location along the first axis, wherein the combination is associated across a plurality of locations along the second axis, and forming a second output in dependence on the operation; forming a representation of the one or more features of the input image in dependence on at least the first output and the second output; and training a convolutional neural network from the representation.

[0020] The identified regions can be regions of the image and / or regions of a tensor describing the image. The tensor can have spatial dimensions and feature or attribute dimensions corresponding to the image.

[0021] The attribute graph can be or include a feature map. The attribute graph can include the input image itself.

[0022] According to a third aspect, there is provided an image processing device for storing a model formed by the above method, the device comprising a processor for receiving a second input image and processing the second input image by means of the model to form an output image.

[0023] The processor can be for processing the second input image by means of the model to perform one of a re-projection operation, a Raw to RGB operation and a tile reordering operation on the second input image. This can allow the processor to improve the quality of the input image.

[0024] The device as described above can be a standalone device in a single housing or a distributed device, e.g. involving multiple computers which can be located at the same or different locations. Such a device can comprise one or more processors for performing the steps described above, and a memory for storing code in a non-transient manner, the code being executed by such a processor to perform the method. BRIEF DESCRIPTION OF DRAWINGS

[0025] The present system will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0026] Figure 1 is a schematic diagram of a CNN for image processing;

[0027] Figure 2a The concept of performing associations across multiple dimensions is illustrated;

[0028] Figure 2b An example of an association performed as shown in Figure 2a is shown;

[0029] Figure 3 One possible embodiment of a tensor self-attention architecture is shown;

[0030] Figure 4 Details of a block of the architecture shown in Figure 3 are shown;

[0031] Figure 5 A flow in one possible embodiment of a tensor self-attention process is shown;

[0032] Figure 6 A standard Bayer pattern color filter array on a sensor is shown;

[0033] Figure 7 It is illustrated that colors are packed into a mosaic;

[0034] Figure 8 Variants of the Unet architecture used to implement the systems described herein are shown;

[0035] Figure 9 Result comparisons for Raw to RGB processing tasks are shown;

[0036] Figure 10 and Figure 11 Results of inpainting tasks are shown. DETAILED DESCRIPTION

[0037] The image processing systems described herein involve extracting information about the intensity of a range of attributes at a plurality of locations in an image. For each attribute, a representation (e.g. a set of data) is formed which represents the intensity of that attribute at a plurality of locations in the image. The locations can be regularly or irregularly spaced. Conveniently, they can correspond to the locations of pixels or blocks of pixels in the image. Each representation can be thought of as an attribute map of the intensity of the corresponding attribute in the image. The representations can be combined into a 3D tensor in which two axes correspond to the spatial axes of the image (conveniently, the horizontal (X) and vertical (Y) axes of the image) and the third axis corresponds to the set of attributes (C). One of the attributes can be thought of as the data of the input image itself; or the image itself can be an X,Y matrix forming a 2D layer in the tensor C. The value in the tensor at an X,Y,C location represents the intensity of attribute C at the X,Y location in the image.

[0038] Non-limiting examples of attributes can include luminance, local contrast or the presence rate of some color or feature (e.g. a face, a person, a vehicle, a sign or an animal).

[0039] The tensor is then processed to detect similarities between the 2D components in the tensor. Each 2D component (“layer”) of the tensor describes a pattern of intensities. These patterns are compared along the third axis of the tensor to form an intermediate comparison output. Importantly, this process is performed for 2D layers which include the C axis of the tensor, and the comparison of the patterns of these layers is performed along the spatial (X or Y) axes of the tensor. This enables other information about the similarity of parts of the image to be gathered: for example, repeating patterns.

[0040] In other words, in a first comparison step, 2D layers having C different from each other at X, Y are compared to detect similarities in their patterns. The comparison detects regions of those layers having similarities at common X, Y locations. An intermediate output of this step is generated. This output can indicate X, Y regions of the image where a plurality of attributes are particularly strong or not strong. In a second comparison step, 2D layers having X or Y different from each other at C are compared to detect similarities in their patterns. For ease of explanation, assume that these 2D layers are at C and X locations, but the necessary modifications can be made to the 2D layers at C and Y when performing the second comparison step. The comparison or correlation operation detects regions of those layers having similarities at common C, X locations. An intermediate output of this step is generated. This output can indicate combinations of X and C that have a common trend in the strength or non-strength of attributes along Y. A third comparison step can be performed in a similar manner on 2D layers at C and other X and Y. The intermediate outputs can then be processed together to obtain information about similarities on the input image.

[0041] Figure 2a This approach is illustrated. An input tensor 1 having dimensions X (alternatively referred to as W), Y (alternatively referred to as H), and C is formed. The input tensor is considered to be composed of sets of 2nd, 3rd, 4th layers located at C, W, H, W, and C, H, respectively. The elements within each layer of each of these sets are compared to each other to identify similarities in the patterns they exhibit at common locations in the respective layer planes. For example, the C, W 2nd layers are compared to each other to identify similarities in their patterns at common locations H. Each of these three comparisons produces a respective intermediate output 5, 6, 7 that represents the commonality in the strength of elements within the layers at locations across the respective axis pair. For example, the intermediate output 5 indicates that the strength or non-strength at C, W locations has commonality. Each intermediate output includes a set of scores that indicate the overall similarity or deviation of values in the respective location tensor 1 in the respective axis pair plane. This approach can describe the complex relationships present in the input tensor. Each intermediate output is a similarity matrix of the respective plane of the input tensor (HWxHW, CHxCH, CWxCW). Conveniently, each point in the matrix can possess a score (e.g., from 0 to 1) that indicates how close the elements in the respective rank of the input tensor orthogonal to the dimension of the similarity matrix are to each other.

[0042] Figure 2b The case of comparing the C, W 2nd layers to each other to identify similarities in their patterns at common locations is illustrated. Each C, W element is described by a series of H attributes, which are then compared. In the figure, the comparison between the 10th and 18th CW elements is illustrated.

[0043] Accordingly, a tensor describing patterns of multiple attributes of an image can be analyzed in multiple dimensions. This process can generate information about the image that can help analyze the image. In one example, a device can receive an input image for processing; analyze the image to detect patterns of multiple attributes in the image, forming an input tensor; analyze the input tensor as described above; and then use the output of this analysis to perform a function such as improving the quality of the image or detecting features in the image. In another example, analyzing tensors in the manner described above can be used to train a machine learning algorithm. In this example, a device can receive a plurality of images in turn, and for each image analyze the image to detect patterns of multiple attributes in the image, forming an input tensor and analyzing the input tensor as described above. The results of this analysis can then be input into a machine learning model. The machine learning model can then generate an adjusted version of the image, which can be tested against a ground truth image (e.g., a version of the corresponding input image with improved quality). Depending on this comparison, the machine learning model can be adjusted. After multiple iterations of this process, the machine learning model can be stored and passed to other devices for use. In each case, the respective data processing steps can be performed by one or more computers programmed with appropriate code executable by the computers, the code stored in a non-transient manner such as in non-volatile memory. The computer or each computer can have one or more processors for executing the code.

[0044] To implement processes such as those described above, a module can be provided that, given an input tensor, uses self-attention information extracted along different dimensions of the tensor to capture complex interdependencies. The extracted self-attention information is combined with the input in such a way that an output tensor is created that has the same dimensions but with higher discriminative power. In one possible embodiment, a machine learning algorithm can be used to extract the self-attention information. In this approach, the proposed self-attention process can be performed according to learned or learnable parameters.

[0045] In summary, to highly exploit the relationships between elements in an input tensor, self-attention is computed on all multiple dimensions of the input tensor, preferably on all dimensions. This approach can capture the associations between channels / attributes compared to previous approaches. This tensor self-attention mechanism can be applied once or multiple times in a deep CNN to improve its performance.

[0046] Self-attention mechanisms can be thought of as mechanisms that identify interconnections or dependencies in the input. A typical self-attention mechanism uses a similarity function, which is usually a real-valued function, that quantifies the similarity between two signals. While there is no single similarity metric definition, in practice such metrics often behave as the inverse of a distance metric: they take relatively large values for similar signals, and zero or negative values for very different signals.

[0047] As noted above, similarity can be identified independently along each dimension of the input tensor. Processing along different dimensions of the tensor can extract similarity not only spatially but also across channels, potentially capturing richer similarity information.

[0048] The input tensor can have different ranges on each of its dimensions, depending on the size and aspect ratio of the input image and the number of channels analyzed. Thus, extracting similarity on multiple dimensions can result in intermediate output matrices of different sizes. It is convenient to fuse these matrices to produce an output tensor of the same size as the input tensor. The resulting output tensor has self-attention rich features. These features are more discriminative than those produced by some other methods, and can produce more accurate outputs.

[0049] When used as a block in a deep neural network, the process of analyzing an input tensor as described herein can be used to solve a variety of computer vision problems. Examples of such problems include inpainting (i.e., filling in regions of an image where data is missing), Raw to RGB mapping, and reconstructing an image from reordered or rearranged portions of the image.

[0050] Figure 3 The high-level structure of the method when used as a deep learning block is described. A deep learning model encoder 10 maps a degraded input 11 to a tensor X. This includes forming a combination of attribute maps of the input tensor. These are processed in a self-attention block 12 in the manner described above at the bottleneck of a CNN. The output of the self-attention block 12 is passed to a decoder 13 that operates on the input image to form an output image 14 based on the output of the block 12.

[0051] Figure 4 The contents of the self-attention block 12 are shown in more detail in one possible embodiment. Figure 5 One possible embodiment of tensor self-attention is shown. On each dimension, the tensor is treated as a set of matrices. A 1x1 convolution kernel is applied to each matrix, followed by a sequence of two matrix multiplications. This or another suitable process implements self-attention on each of these sets of matrices. These are combined to form an output tensor Z. This method can be used for a variety of problems, including inpainting, Raw to RGB, and reconstructing rearranged inputs. In Figure 5In the illustrated system, self-attention is embodied as a matrix multiplication. Other operations can be used instead.

[0052] Given an N-order dimensional tensor, the process described herein applies N parallel and independent self-attention mechanisms to extract different information from the same input. The process then fuses their results together with the original input tensor.

[0053] Matricization, also known as unfolding or flattening, is the process of reordering the elements of an N-way array into a matrix. For example, a 2 x 3 x 4 tensor can be arranged as a 6 x 4 matrix or a 3 x 8 matrix, and so on. The n-mode matricization of a tensor is denoted as X(n) and arranges the n-mode fibers of X as columns of the resulting matrix.

[0054] The input tensor X is a 3D tensor representation of a 2D input image. It is extracted using a CNN module (e.g., the encoder 10 of Figure 3 The tensor self-attention block 12 takes X as input and outputs its enriched representation Z. Using the present method allows the subsequent decoder module 13 to obtain a higher quality output image.

[0055] The input tensor X of dimension X x Y x C is unfolded over its 3 modes. In other words, it is rearranged into 3 distinct 2-D matrices. Each group of matrices concentrates on a different slice of the input. Given the tensor representation, a self-attention module is applied to each mode separately. All self-attention outputs are then combined with the input tensor to produce the output.

[0056] In CNNs, convolutional operations are the building blocks that process one local neighborhood at a time; therefore, when these operations are applied repeatedly, long-range dependencies can be captured. This comes with some limitations, such as low computational efficiency and optimization difficulties. To help address this issue, the present method computes useful complex interdependencies of the input tensor.

[0057] Figure 5 The self-attention module is shown in more detail. In Figure 5 The “+” symbol denotes summation, the “x” symbol denotes matrix multiplication, and the arrows denote inputs, where there can be modulation by learnable scalars. At the indicated locations, the rectangles represent 1 x 1 convolutional operations. This module implements a self-similarity module that performs the following steps:

[0058] (1) Unfold the N-order input tensor over its respective N modes and use a convolutional operator to embed each mode into a separate learned subspace.

[0059] (2) Using a matrix multiplication operator, the response of each element given all other elements is computed for each nth embedded modality (X, Y, C). In this way, the method processes all possible pairs and computes a similarity score for each pair, generating an n-modality attention map. Another matrix multiplication using the original nth modality input integrates this similarity information in the output features. This process is described by the following equation:

[0060] (3) The output of the self-attention of each modality is added through a residual connection and added to the original input features. This can enhance the discriminability of the original input tensor.

[0061] An example of how to use this module will now be described. In this example, the module is used in a deep learning process to perform Raw to RGB encoding. This non-limiting embodiment of the method is based on deep learning (e.g. using a CNN). This stage has the raw data as input. The raw data passed as input can be an image formed using a colour filter array (CFA) that captures a particular colour of light at each pixel, for example using the well-known Bayer pattern shown in Figure 6 This pattern has a repeating 2x2 mosaic that is tiled across the image. The 2x2 mosaic includes one blue element, two green elements and one red element. The raw data captured typically has a large dynamic range: for example, 10-bit data can represent 1024 different levels of each red, green or blue colour. An image captured in this format is referred to as a mosaic.

[0062] The mosaic image can be packed into four colour channels representing red, first green, second green and blue as shown in Figure 7 In the packed format, the spatial resolution of each colour channel is half that of the original mosaic image resolution.

[0063] The method applies a convolutional neural network to process the mosaic image. The CNN learns a set of filters that are applied to the image through convolution. The convolution is designed to be spatially invariant, meaning that the convolution has the same effect when applied anywhere in the image. When applying the convolution on the mosaic image, it is desirable for the convolution to remain spatially invariant despite the design of the CFA (e.g. when the filter is centred on a blue pixel, it can have a different effect than when it is centred on a red pixel). One simple way to achieve this is to pack the data into channels of similar colour and then each channel can be processed using a spatially invariant convolution in the CNN.

[0064] Figure 8An example of a suitable CNN design is shown. The network takes the original single channel input 20, packs the data into four channels 21, and then processes it with a Unet. This fully convolutional network uses an encoder-decoder architecture with skip connections. Between the encoder 22 and the decoder 23 of the network, a tensor self-attention block 24 (e.g. of the type described above) integrates information about self-similarity.

[0065] The encoder part 22 processes the original input with five successive layers. Each layer applies two sets of 3x3 convolutional filters (and a ReLU activation function) and a “max-pooling” operation to its input. The first convolution doubles the number of filters (i.e. channels). The max-pooling operation reduces the spatial image resolution by half (i.e. from X, Y, C to X / 2, Y / 2, C). The image is processed at multiple scales, and the network adapts to different frequency content. This generates output channels that capture features inherent in the data and relevant for the Raw-to-RGB task.

[0066] As mentioned above, the tensor self-attention module 24 is used to compute self-attention over the input tensor. It takes as input the encoder features (X / 32, Y / 32, 512) and produces as output a matrix of the same dimensions.

[0067] The decoder part 23 processes the output of the tensor self-attention block with four successive layers of two sets of 3x3 convolutional filters and a transposed convolution operation. The transposed convolution is an up-sampling layer that doubles the spatial resolution in each dimension (width and height) and halves the number of filters. The input to each layer is a concatenation of (i) high-resolution features from the encoding part related to the same spatial resolution and (ii) the output of the previous decoding layer (i.e. spatially up-sampled features). After multiple iterations, the subsequent two convolutions learn to combine more accurate outputs from the concatenated inputs.

[0068] During training, the network learns the convolutional filters. This is done using training pairs, each consisting of an input Raw image and a corresponding reference RGB image, which is used as ground truth (GT). Initially, the convolutional filters are set to random values. The mosaic input Raw image is input into the network, which regresses an output image that is a candidate RGB output representing the input image. The difference between the regressed output image and the GT image forms an error, which is back-propagated through the network from output to input using gradients. The weights of the network are then updated to reduce the error. The training process is iterated using a large set of image pairs until the network weights converge appropriately. Once the network is trained, it can be applied to arbitrary Raw inputs to recover their RGB channels.

[0069] Figure 9 to Figure 11Results for an example system using the present method are shown.

[0070] Figure 9 The ground truth image corresponding to the example Raw input is shown on the right, and the image formed using the present method is shown on the left. Figure 8 The network of the type shown in Fig. 2 utilizes the present self-attention module to form RGB images, and for comparison in the middle, the RGB images formed using the Raw-to-RGB method of “Chen et al.” (Chen, Chen, Xu, and Koltun, “Learning to See in the Dark,” IEEE Conference on Computer Vision and Pattern Recognition, 3291-3300, 2018) are shown. The image on the left estimates sharper edges and more realistic colors compared to the image in the middle.

[0071] Figure 10 An example of the present method is shown. Figure 8 The network of the type shown in Fig. 2 utilizes the present self-attention module to form RGB images, and for comparison in the middle, the RGB images formed using the Raw-to-RGB method of “Chen et al.” (Chen, Chen, Xu, and Koltun, “Learning to See in the Dark,” IEEE Conference on Computer Vision and Pattern Recognition, 3291-3300, 2018) are shown. The image on the left estimates sharper edges and more realistic colors compared to the image in the middle. Figure 10 The image on the left is missing a region. When these images are input to a trained network, the output is as shown on the right. Figure 10

[0072] Figure 11 The results of the present method and prior art methods are shown. The ground truth image is on the right. The first column is formed using the present method. Figure 8 The network of the type shown in Fig. 2 utilizes the present self-attention module to form RGB images, and for comparison in the middle, the RGB images formed using the Raw-to-RGB method of “Chen et al.” (Chen, Chen, Xu, and Koltun, “Learning to See in the Dark,” IEEE Conference on Computer Vision and Pattern Recognition, 3291-3300, 2018) are shown. The image on the left estimates sharper edges and more realistic colors compared to the image in the middle.

[0073] The applicant hereby discloses separately each and every feature described herein and any combination of two or more such features. In view of the foregoing description, those of ordinary skill in the art will understand that the features or combinations of features disclosed herein can be employed in any number of arrangements without departing from the scope of the application. Accordingly, the applicant expressly reserves the right to amend the claims to specifically claim any of the features or combinations of features disclosed herein, in any combination with one another, and / or to claim any other features or combinations of features that are within the scope of the application.​

Claims

1. An image processing apparatus for recognizing one or more features of an input image, characterized in that, The device includes a processor, the processor being used for: Receive the input image, which extends along a first axis and a second axis; A series of attribute maps (2, 3, 4) are formed based on the received input image, and each attribute map represents the intensity of the corresponding attribute at multiple locations in the image; A first correlation operation is performed by identifying regions that are related to patterns in multiple attribute graphs in the series of attribute graphs, and a first output is formed based on the operation (6); Perform a second correlation operation to identify combinations of (i) attributes and (ii) portions of the image that have a common position along the first axis, wherein the combinations are correlated across multiple positions along the second axis, and form a second output (5, 7) based on the operation; and A representation of one or more features of the input image is formed based at least on the first output and the second output; Wherein, one of the first axis and the second axis is the horizontal axis X of the image, and the other of the first axis and the second axis is the vertical axis Y of the image. The attributes form a set C, and the image and the attribute graph together form a tensor with dimensions C, X, and Y.

2. The image processing apparatus according to claim 1, characterized in that, The processor is used for: A third correlation operation is performed to identify combinations of (i) attributes and (ii) portions of the image that have a common position along the second axis, wherein the combinations are correlated across multiple positions along the first axis, and a third output is formed based on the operation; The representation of one or more features forming the input image further depends on the third output.

3. The image processing apparatus according to claim 1, characterized in that: The output of the first correlation operation is a similarity matrix of dimensions X and Y; The output of the second correlation operation is a similarity matrix of dimension C and one of X and Y.

4. The image processing apparatus according to any one of claims 1 to 3, characterized in that, The attributes include one or more of the following: the presence rate of a specific hue, brightness, local contrast, and a determined representation of the local likelihood of a specific feature.

5. The image processing apparatus according to claim 4, characterized in that, The feature in question is the face.

6. The image processing apparatus according to claim 4 or 5, characterized in that, The processor is used to perform feature recognition operations on the input image to form a graph that includes local likelihood estimates of specific features at multiple locations in the input image.

7. The image processing apparatus according to any one of claims 1 to 6, characterized in that, The processor is used to train a convolutional neural network based on the representation.

8. A method for recognizing one or more features of an input image, characterized in that, The method includes: Receive the input image, which extends along a first axis and a second axis; A series of attribute maps (2, 3, 4) are formed based on the received input image, and each attribute map represents the intensity of the corresponding attribute at multiple locations in the image; A first correlation operation is performed by identifying regions that are related to patterns in multiple attribute graphs in the series of attribute graphs, and a first output is formed based on the operation (6); A second correlation operation is performed to identify combinations of (i) attributes and (ii) portions of the image that have a common position along the first axis, wherein the combinations are correlated across multiple positions along the second axis, and a second output (5, 7) is formed based on the operation; A representation of one or more features of the input image is formed based at least on the first output and the second output; and Train a convolutional neural network based on the representation; Wherein, one of the first axis and the second axis is the horizontal axis X of the image, and the other of the first axis and the second axis is the vertical axis Y of the image. The attributes form a set C, and the image and the attribute graph together form a tensor with dimensions C, X, and Y.

9. An image processing apparatus for storing a model formed by the method of claim 8, characterized in that, The device includes a processor for receiving a second input image and processing the second input image using the model to form an output image.

10. The image processing apparatus according to claim 9, characterized in that, The processor is used to process the second input image with the aid of the model to perform one of the following operations on the second input image: redrawing, Raw to RGB, and tile reordering.

Images