Method and device for enhanced image processing using vision graph neural network

Abstract

It is described a method for processing an image using a vision graph neural network, said vision graph neural network comprising a window-based grapher module (7) including a first fully connected layer with batch normalization (9), a windows partitioning module (10), a dynamic graph convolution module (11), a windows reverse module (12), a second fully connected layer with batch normalization (13) and a skip connection (15), wherein said window-based grapher module (7) is configured to: - process a feature vector (X) of said image (2) through said first fully connected layer with batch normalization (9) to obtain a normalized feature vector; - partition said normalized feature vector into a plurality of non-overlapping windows using said windows partitioning module (10); - for each window, construct a graph where nodes represent patches of said image (2) within the respective window and edges represent relationships between said nodes, and apply a graph convolutional operation to each graph to update node features within each window using said dynamic graph convolution module (11); - reshape the updated node features from each window back into the format of said normalized feature vector using said windows reverse module (12); - process the reshaped feature vector through said second fully connected layer with batch normalization (13); combine said feature vector (X) directly with an output of said second fully connected layer with batch normalization (13) using said skip connection (15).

Description

[0001] -SVT079-PC00-

[0002] METHOD AND DEVICE FOR ENHANCED IMAGE PROCESSING USING

[0003] VISION GRAPH NEURAL NETWORK

[0004] DESCRIPTION

[0005] The present invention relates to a method and a device for enhanced image processing using vision graph neural network.

[0006] In the last decade, the field of computer vision has progressed significantly, largely due to the success of deep neural networks (see Yann LeCun, Yoshua Bengio and Geoffrey Hinton, "Deep learning", Nature, 521 (7553) :436-444, 2015, reference

[0007]

[0020] ).

[0008] These models are now established as state-of-the-art in several tasks such as image classification, object detection, semantic segmentation, and so on (see Johannes Balle, Valero Laparra, and Eero P. Simoncelli, "End-to-end optimized image compression", ICLR, 2017, reference [1]; Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko, "End-to-end object detection with transformers", ECCV, 2020, reference [3]; Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, "Deep residual learning for image recognition", CVPR, 2016, reference

[0014] ; Joseph Redmon, Santosh Diwala, Ross Girshick and Ali Farhadi, "You only look once: Unified, real-time object detection", CVPR, 2016, reference

[0030] ; Olaf Ronneberger, Philipp Fischer, and Thomas Brox, "U-Net: Convolutional networks for biomedical image segmentation", MICCAI, 2015, reference

[0032] ).

[0009] In particular, Convolutional Neural Networks (CNNs) (see Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner, "Gradient-based learning applied to document recognition", Proceedings of the IEEE, 86(ll):2278-2324, 1998, reference

[0010]

[0021] ) exploit the locality of natural images to extract features while Vision Transformers (ViTs) (see Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Geliy, Jakob Uszkoreit, and Neil Houlsby, "An image is worth 16x16 words: Transformers for image recognition at scale", ICLR, 2021, reference [6]; Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, -SVT079-PC00-

[0011] Stephen Lin, and Baining Guo, "Swin transformer: Hierarchical vision transformer using shifted windows", ICCV, 2021, reference

[0024] ; Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou, "Training data-efficient image transformers & distillation through attention", ICML, 2021, reference 36]) implement the attention operator to exploit long-range dependencies of the input image.

[0012] Recently, vision-based Graph Neural Networks (GNNs) (see Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu, "Vision GNN: An image is worth graph of nodes", NeurlPS, 2022, reference

[0011] ; Mustafa Munir, William Avery, and Radu Marculescu, "Mobilevig: Graph-based sparse attention for mobile vision applications", CVPR Workshop, 2023, reference

[0027] ; Mustafa Munir, William Avery, Md Mostafijur Rahman, and Radu Marculescu, "Greedyvig: Dynamic axial graph construction for efficient vision gnns", CVPR, 2024, reference

[0028] ), have been proposed with promising results for vision tasks.

[0013] Vision GNNs first build a graph over the features extracted from the image, and then apply graph convolutions instead of regular 2D convolutions (CNNs) or the self-attention mechanism (ViTs). As a result, vision GNNs have benefited from the rich literature of GNNs (see Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip, "A comprehensive survey on graph neural networks", IEEE T-NNLS, 32(l):4-24, 2020, reference

[0043] ), adding a new dimension to the landscape of deep learning for image analysis.

[0014] Despite the promising results achieved by vision GNNs, there still remain some open challenges. More precisely, the Vision GNN (ViG) model (see reference

[0011] ) relies on the k-Nearest Neighbors (k-NN) method to construct the graph. Therefore, the computational complexity of ViG increases quadratically with the number of nodes (patches) extracted from the image and hence with the image size (refer to Figures 6a and 6b for further details). This hinders their applicability to real- world large-scale datasets and high-resolution images, limiting their practical use. Overcoming the scalability issue in vision GNNs is crucial for their wider adoption and deployment in real-world applications.

[0015] It follows an overview of deep learning networks typically adopted in -SVT079-PC00- computer vision like CNNs and ViTs. Then GNNs are reviewed, analyzing their applications to visual tasks and their limitations.

[0016] CNNs and ViTs

[0017] Convolutional Neural Networks (CNNs) started dominating the computer vision field since the seminal AlexNet paper (see Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, "ImageNet classification with deep convolutional neural networks", NeurlPS, 2012, reference

[0019] ). CNNs exploit the locality of pixels to extract features from the input image useful to the task on which they are trained. CNNs represented the de-facto standard to solve different tasks from image classification to object detection (see reference

[0030] ; and Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, "Faster R-CNN: Towards real-time object detection with region proposal networks", NeurlPS, 2015, reference

[0031] ), semantic segmentation (see Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick, "Mask R-CNN", ICCV, 2017, reference

[0013] , and reference

[0032] ), image compression (see reference [1] and David Minnen, Johannes Balle, and George D Toderici, "Joint autoregressive and hierarchical priors for learned image compression", NeurlPS, 2018, reference

[0026] ), and many others. The rapid development that these architectures have experienced over the past decade has led to the development of models such as ResNet (see reference

[0014] ) and MobileNet (see Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam, "MobileNets: Efficient convolutional neural networks for mobile vision applications", ArXiv, 2017, reference

[0016] ) among others (see G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks", CVPR, 2017, reference

[0017] ; C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions", CVPR, 2015, reference

[0034] ; Mingxing Tan and Quoc Le, "EfficientNet: Rethinking model scaling for convolutional neural networks", ICML, 2019, reference

[0035] ).

[0018] More recently, researchers in computer vision have focused on Visual Transformers (ViTs), with the self-attention mechanism at its core. ViTs build upon the attention mechanism proposed by the Transformer architecture (see Ashish -SVT079-PC00-

[0019] Vaswani, Noam Shazeer, Niki Pa rm ar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Ulia Polosukhin, "Attention is all you need", NeurlPS, 2017, reference

[0037] ) for Natural Language Processing (NLP) tasks in origin, and later on applied with success to different computer vision tasks (see reference [3]; reference [6]; reference

[0024] ; and Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He, "Non-local neural networks", CVPR, 2018, reference

[0040] ).

[0020] Attention enables capturing long-range dependencies between pixels, achieving state-of-the-art results in several computer vision tasks. The Swin Transformer (see reference

[0024] ), in particular, proposed a hierarchical Transformerbased architecture to extract tokens at different scales and work with high-resolution images. The multi-head self-attention operator in Swin Transformer is computed in non-overlapped windows.

[0021] To introduce cross-window connections, the authors proposed a shifted window partitioning approach that alternates with the regular window partitioning in consecutive blocks of Swin Transformer. This method allows for connections between neighboring windows in the previous layer, leading to improvements in image classification, object detection, and semantic segmentation (see reference

[0024] ).

[0022] The core function in Swin Transformers can be thought of as an attention operator applied in fully connected graphs from windows in images.

[0023] Graphs in Computer Vision

[0024] GNNs emerged as an extension of the convolution operation of CNNs for regular-structured data such as images to the graph domain. GNNs are typically used for learning graph-structured data representations.

[0025] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann Le-Cun, "Spectral networks and locally connected networks on graphs", ICLR, 2014, reference [2], proposed the first modern GNN by extending the convolutional operator of CNNs to graphs. Incorporating concepts of signal processing on graphs, M. Defferrard, X. Bresson, and P. Vandergheynst, "Convolutional neural networks on graphs with fast localized spectral filtering", NeurlPS, 2016, reference [5]) introduced localized spectral filtering for graphs.

[0026] Later, Thomas N Kipf and Max Welling, "Semi-supervised classification with -SVT079-PC00- graph convolutional networks", ICLR, 2017, reference

[0018] , approximated the spectral filtering operation to obtain efficient Graph Convolutional Networks (GCNs). Inspired by these works, Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio, "Graph attention networks", ICLR, 2018, reference

[0038] , presented the attention mechanism on GNNs, resulting in a graph where the connection weights are unique and learned for each edge. This allows GATs (Graph Attention Networks) to effectively model complex relationships between nodes, although with increased computational complexity.

[0027] Even though GNNs have generally been adopted for graph-based data, see reference

[0018] , they have recently demonstrated remarkable success when applied to tasks such as image classification (see references

[0011] ,

[0027] ,

[0028] ) and segmentation (see Jhony H Giraldo et al., "Hypergraph convolutional networks for weakly- supervised semantic segmentation", ICIP, 2022, reference [8]).

[0028] The Vision GNN (ViG) model (see reference

[0011] ), in particular, drew inspiration from the partition concept introduced in ViTs (see reference [6]), dividing the input image into smaller patches, and considering each of these patches as a node in the graph of the image. To establish connections between these nodes, the k-NN algorithm is adopted by considering the similarity of nodes in the feature space. These features are updated using graph convolution operators, considering the features of the node itself and those of its neighbors (see Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl, "Neural message passing for quantum chemistry", 2017, reference [7]). Following a similar paradigm as Transformers, these features contribute to the classification of the entire graph, thereby classifying the entire image.

[0029] Using a graph can be beneficial in image processing tasks as it allows for the exploitation of non-local dependencies without the need for multiple convolutional layers and to model complex objects having irregular shapes.

[0030] Furthermore, graphs are a more general data structure with respect to a grid of pixels (as modeled by GNNs) or a fully connected graph of patches (as modeled by ViTs). These advantages have led graph-based models to reach state-of-the-art not only in image classification but also in object detection and instance segmentation -SVT079-PC00-

[0031] (see references

[0011] and

[0028] ). However, vision GNNs still have a very high computational complexity, especially when working with high-resolution images. For this reason, in the first layers of ViG, the graph is constructed in a bipartite way, connecting patches of the original feature maps with a subsample version of it, obtained using a non-learnable subsampling filter. Although this approach decreases the complexity of ViG, it still has a quadratic time complexity, making it slow for processing large images.

[0032] More recently, new techniques have emerged to reduce the complexity of ViG by focusing on the graph construction phase. MobileViG (see reference

[0027] ), for instance, proposed a Sparse Vision Graph Attention (SVGA) module, in which the graph is statically constructed, thus without adopting k-NN. Here a patch of the image is connected to patches at a certain hop distance on the same row and column. In this way the number of connections depends on the size of the image, rapidly increasing the memory required to perform graph convolutions. Thus, to obtain a mobile-friendly model, MobileViG only adopts the SVGA module in the last stage of the architecture (where the input tensor is smaller) while the previous stages are implemented using classical depth-wise 2D convolutions. This results in a hybrid model in which the GNNs and GNNs are both adopted. GreedyViG (see reference

[0028] ) proposed a dynamic version of SVDA named Dynamic Axial Graph Construction (DAGC). This module starts from the same fixed graph of SVDA and dynamically masks some connections. To create this mask, GreedyViG estimates the mean and standard deviation of the Euclidean distances between the patches in the original image and a diagonally flipped version. Moreover, this DAGC module is implemented in each stage of the architecture, making this CNN-GNN model highly memory-intensive.

[0033] The present invention aims at solving these and other problems by providing a method and a device for efficient image processing using vision graph neural network, resulting in computational complexity that grows linearly with the image size.

[0034] It is a further scope of the present invention to provide a method and a device for enhanced image processing using vision graph neural network that significantly -SVT079-PC00- reduces memory usage, making it feasible to process high-resolution images.

[0035] It is a further scope of the present invention to provide a method and a device for enhanced image processing using vision graph neural network that allows for efficient scaling, enabling the application of vision GNNs to larger datasets and higher resolution images without excessive resource consumption.

[0036] It is a further scope of the present invention to provide a method and a device for enhanced image processing using vision graph neural network that reduces the complexity of the graph construction, making the process more manageable and efficient.

[0037] It is a further scope of the present invention to provide a method and a device for enhanced image processing using vision graph neural network that captures relevant features by focusing on localized regions within the image, enhancing efficiency and performance.

[0038] It is a further scope of the present invention to provide a method and a device for enhanced image processing using vision graph neural network that mitigates over-smoothing in GNN Layers to encourage feature diversity and maintain discriminative power.

[0039] It is a further scope of the present invention to provide a method and a device for enhanced image processing using vision graph neural network that avoids missing out on global information and long-range dependencies within the image.

[0040] It is a further scope of the present invention to provide a method and a device for enhanced image processing using vision graph neural network that provides for dynamic adaptation to image borders.

[0041] It is a further scope of the present invention to provide a method and a device for enhanced image processing using vision graph neural network that performs well on both standard benchmark datasets and high-resolution downstream tasks.

[0042] It is a further scope of the present invention to provide a method and a device for enhanced image processing using vision graph neural network that provides a robust, scalable, and efficient solution for image processing using vision GNNs, making it suitable for a wide range of real-world applications. -SVT079-PC00-

[0043] In a nutshell, to address the scalability challenges of vision GNNs, it is proposed a novel Windowed vision Graph neural Network (WiGNet) model.

[0044] Similar to previous vision GNNs, WiGNet treats an image as a graph, yet in a fundamentally different manner. Namely, the image is first partitioned into non-overlapping windows and only then a separate graph is built for each window as shown in Figure 1. The complexity of building the graphs in the method according to the invention grows only linearly with the number of windows, while maintaining competitive results in image classification tasks. By focusing on localized regions within the image through windowed processing, WiGNet efficiently captures relevant features while alleviating the scalability issues encountered by previous vision GNN approaches.

[0045] WiGNet is, in essence, different from Swin Transformers since (i) it operates in k-NN graphs instead of fully connected graphs, and (ii) a GNN function is used instead of the self-attention mechanism. These two changes achieve competitive results regarding the Swin Transformer, other ViTs, and CNN models.

[0046] The method according to the invention makes the following significant contributions:

[0047] • to the best of the Applicants' knowledge, it is the first time that the concept of windowed processing is introduced in the context of vision GNNs;

[0048] • it is shown that the computational and memory complexity of WiGNet only grows linearly with the image size, paving the way for broader applications of graph-based models in computer vision;

[0049] • the method according to the invention was thoroughly validated in the ImageNet-lk benchmark dataset and its adaptability tested as a feature extractor on CelebA-HQ as a downstream task with higher resolution images. In both of these scenarios, WiGNet outperforms or obtains competitive performance regarding previous deep learning models like GNNs, ViTs, and ViGs.

[0050] The Applicants' results in ImageNet-lk suggest that WiGNet successfully exploits vision GNNs for image classification tasks. In addition, classification results of higher-resolution images show that WiGNet is able to achieve state-of-the-art results while keeping complexity under control. -SVT079-PC00-

[0051] Unlike the prior art methods, WiGNet always works on the original feature maps and not on an undersampled version (like ViG). The method according to the invention partitions these feature maps into windows of fixed size in which the graph can be constructed. Moreover, the k-NN operator is not replaced with a fixed graph structure, but the locality of pixels is exploited to reduce the complexity of this operation, making it linear with respect to the size of the input image.

[0052] According to the invention, it is therefore described a method for processing an image using a vision graph neural network, the vision graph neural network comprising a window-based grapher module including a first fully connected layer with batch normalization, a windows partitioning module, a dynamic graph convolution module, a windows reverse module, a second fully connected layer with batch normalization and a skip connection, wherein the window-based grapher module is configured to: process a feature vector of the image through the first fully connected layer with batch normalization to obtain a normalized feature vector; partition the normalized feature vector into a plurality of non-overlapping windows using the windows partitioning module; for each window, construct a graph where nodes represent patches of the image within the respective window and edges represent relationships between the nodes, and apply a graph convolutional operation to each graph to update node features within each window using the dynamic graph convolution module; reshape the updated node features from each window back into the format of the normalized feature vector using the windows reverse module; process the reshaped feature vector through the second fully connected layer with batch normalization; combine the feature vector directly with an output of the second fully connected layer with batch normalization using the skip connection.

[0053] According to an aspect of the method for processing an image using a vision graph neural network according to the invention, the graph convolutional operation applied in the dynamic graph convolution module comprises using a k-Nearest Neighbors algorithm to determine the edges between the nodes based on feature similarity.

[0054] According to a further aspect of the method for processing an image using a -SVT079-PC00- vision graph neural network according to the invention, the k-Nearest Neighbors algorithm dynamically adjusts a number of neighbors considered for each node based on the position of the node within the respective window.

[0055] According to a further aspect of the method for processing an image using a vision graph neural network according to the invention, the graph convolutional operation comprises a Max-Relative graph convolution, which aggregates features from neighboring nodes and updates the node features using a learnable weight matrix.

[0056] According to a further aspect of the method for processing an image using a vision graph neural network according to the invention, the windows partitioning module is configured to partition the normalized feature vector into windows of a fixed size.

[0057] According to a further aspect of the method for processing an image using a vision graph neural network according to the invention, it is provided to aggregate the combined feature vector and the output of the second fully connected layer with batch normalization to form a global feature representation of the image.

[0058] According to a further aspect of the method for processing an image using a vision graph neural network according to the invention, it is provided to classify the image based on the global feature representation of the image.

[0059] According to a further aspect of the method for processing an image using a vision graph neural network according to the invention, it is provided to apply a shifting operation to the plurality of non-overlapping windows in the windows partitioning module to create shifted windows.

[0060] According to a further aspect of the method for processing an image using a vision graph neural network according to the invention, the shifting operation involves: cyclically shifting portions of the normalized feature vector to ensure that each window overlaps with adjacent windows; and using a masking mechanism to control the connections between nodes in the shifted windows to ensure that only adjacent nodes are connected.

[0061] According to a further aspect of the method for processing an image using a vision graph neural network according to the invention, the cyclically shifting -SVT079-PC00- operation involves copying the top-left part of the normalized feature vector to the bottom-right part to create the shifted windows.

[0062] According to a further aspect of the method for processing an image using a vision graph neural network according to the invention, the dynamic graph convolution module recalculates the graph structure and updates the node features based on the shifted windows to capture cross-window connections.

[0063] According to the invention, it is further described a device for processing an image using graph-based neural network operations, comprising: a first fully connected layer with batch normalization configured to receive a feature vector of the image and to process it to obtain a normalized feature vector; a windows partitioning module configured to partition the normalized feature vector into a plurality of nonoverlapping windows; a dynamic graph convolution module configured to, for each window, construct a graph where nodes represent patches of the image within the respective window and edges represent relationships between the nodes, and apply a graph convolutional operation to each graph to update node features within each window; a windows reverse module configured to reshape the updated node features from each window back into the format of the normalized feature vector; a second fully connected layer with batch normalization configured to process the reshaped feature vector; a skip connection configured to combine the feature vector directly with an output of the second fully connected layer with batch normalization.

[0064] According to the invention, it is further described a modular block comprising the device for processing an image using graph-based neural network operations and a Feed Forward Network module.

[0065] According to the invention, it is further described a vision graph neural network comprising: a stem block for extracting features from an image; at least one modular block followed by a respective downsampling block; a fully connected layer for producing class scores and classifying the image.

[0066] The invention will be described in detail hereinafter through non-limiting embodiments with reference to the attached Figures, wherein:

[0067] Figure 1 shows an implementation of Vision GNN (ViG) according to the prior art and an implementation according to the present invention (WiGNet); -SVT079-PC00-

[0068] Figure 2a shows an embodiment of a vision graph neural network architecture according to the invention;

[0069] Figure 2b shows a graphical illustration of a module of the architecture of Figure 2a;

[0070] Figure 3 shows a table of detailed settings of three versions of WiGnet architecture according to the invention;

[0071] Figure 4 shows an illustrative example of a dynamic graph convolution block according to the invention;

[0072] Figure 5 shows an overview of a cycling operation used to obtain shifted windows;

[0073] Figures 6a and 6b show plots representing a comparison of the computational complexity and the memory complexity, respectively, required by the method according to the present invention with respect to prior art methods;

[0074] Figure 7 shows a table representing the results obtained with the method according to the present invention with respect to other deep learning methods and ImageNet dataset;

[0075] Figure 8 shows a plot representing a comparison between the model according to the present invention and other graph-based prior art models with regard to the number of MACS used and the memory consumption;

[0076] Figure 9 shows a table representing an ablation study in the impact of the shifting operation for the method according to the invention on ImageNet dataset;

[0077] Figure 10 shows a table representing an ablation study in the impact of the shifting operation and the adaptive / (-NN strategy for higher resolution images from the CelebA-HQ dataset;

[0078] Figure 11 shows a table representing ImageNet results using different convolutional layers, where the comparison is performed on the tiny model size without the shifting operation.

[0079] Figure 1 represents an implementation of Vision GNN (ViG), see reference

[0011] , and an implementation according to the present invention (WiGNet). The model according to the present invention first divides images into local windows where graphs are built. This fundamental change dramatically increases computational and -SVT079-PC00- memory efficiency in vision tasks.

[0080] It is now described a Windowed Vision Graph Neural Network (WiGNet) architecture according to the invention. Firstly, some background on graphs and GNNs is provided. Secondly, the architectural design is described. Then the implication of computational complexity is discussed, comparing WiGNet with the ViG model. Finally, three different WiGNet versions that are used for the Applicants' experiments are proposed.

[0081] Preliminaries

[0082] Graph. A graph is a mathematical entity that can be represented as G = (V, £), where V = {1, . . . , N} is the set of N nodes, and S £ {( / zy) | j e V and i / j} is the set of edges between nodes i and j. It is possible to associate F-dimensional feature vectors to every z-th node in G such that x;G RF. Therefore, the whole set of features in G is represented with the matrix X = [Xl, X2, . . . , XN ]TG RNXF.

[0083] Message passing function. In GNNs, the message-passing function is the standard paradigm for computing graph convolutions (see reference [7]). Let x / be the output of a generic graph convolution, the message-passing function can thus be defined as follows:

[0084] V :::• UPDATE : X : . A€G( ( X V j € . V } P , (I) where Ni is the set of neighbors of z, AGG(-) is a generic function used to aggregate neighbor information, and UPDATE(-) updates the representation of the node itself. A graph convolutional layer can be thought of as an implementation of this messagepassing operator by concretely defining the update and aggregation functions.

[0085] WiGNet architecture

[0086] Architecture overview. Figure 2a shows a bird's eye view of the WiGNet architecture 1, implementing a four-stage pyramidal feature extractor, where at each stage features of increasingly smaller sizes are extracted.

[0087] A WiGNet architecture 1 comprises three basic building blocks: (i) a Stem block 3, (ii) a WiGNet block 4, and (iii) a downsampling block 5.

[0088] The Stem block 3 is a simple feature extractor composed of three convolutional layers that receive as input an image 2 of size H x W x 3, divides the image into N -SVT079-PC00- patches and transforms them into a feature vector x;G RFfor each patch, obtaining X = [Xl, X2, . . . , XN ]T.

[0089] The WiGNet block 4 comprises a Window-based Grapher module 7 and a Feed Forward Network (FFN) module 8. The Window-based Grapher module 7 partitions the image into non-overlapping windows, builds a graph for each window, and then local GNN updates are applied to each window.

[0090] This is a fundamentally different approach than ViG (see reference

[0011] ) where a large graph is built on top of the entire image, with the complexity implications discussed above. The FFN module 8 further encourages feature diversity. The downsampling block 5 reduces the feature dimension by merging node representations. Each of the downsampling blocks 5 reduces the number of nodes by a factor of 2 while increasing the size of the feature vectors associated with the remaining nodes.

[0091] The complete network architecture is composed of a stack of the Stem block 3, at least one, preferably four, WiGNet block 4 followed by at least one, preferably four, respective downsampling block 5 (see Figure 3).

[0092] Figure 2a shows a final fully connected layer for producing class scores. Three versions of WiGNet are proposed in Figure 3: (i) tiny (WiGNet-Ti), (ii) small (WiGNet-S), and (iii) medium (WiGNet-M), wherein D represents the feature dimension, E represents the hidden dimension ratio in the FFN, k represents the number of neighbors in GCN (Graph Convolutional Network), W represents the window size, and Hx W represents the input image size.

[0093] The Window-based Grapher module 7 illustrated in Figure 2b is at the core of WiGNet. Preliminary, the feature vector X generated from the Stem module 3 (or the previous Grapher module 7) is processed by a fully connected layer with batch normalization 9. Firstly, a Windows Partitioning module 10 splits the input tensor, or normalized feature vector, into non-overlapping windows. The non-overlapping windows may have a fixed size of M x M. Secondly, the Dynamic Graph Convolution module 11 builds a graph and performs graph convolution independently for each window. The Dynamic Graph Convolution module 11 is in turn at the core of the Window-based Grapher module 7 and is described in detail -SVT079-PC00- later.

[0094] Next, the Windows Reverse module 12 reshapes the output of the Dynamic Graph Convolution module 11 into the original feature vector as generated by the Windows Partitioning module 10. The feature vector is then passed as input to a fully connected layer with batch normalization 13. Finally, the Window-based Grapher module 7 is completed with a skip connection 15.

[0095] Dynamic Graph Convolution component 11

[0096] For each w-th window, the dynamic graph convolution component 11 implements the / (-NN algorithm to produce a graph Gw= (Vw, Ew), where Sw£ {(Z / y) |Z / j e yw} isthe set of edges between nodes i and j and Vwis the set of nodes in Gw. In particular, two nodes (z , / ) are connected if j E N™ , where N™ is the set of k nearest neighbors for the node i belonging to the same window w.

[0097] Therefore, similarly to ViG (see reference

[0011] ), it is applied the Max-Relative graph convolution proposed by Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem, "DeepGCNs: Can GCNs go as deep as CNNs?", ICCV, 2019, reference

[0023] , to update the representation of the z-th node in the w-th window as in formula (2): where || is the concatenation function, and Wupdate is a matrix of learnable parameters.

[0098] Figure 4 shows an example of the computation of x'™ in formula (2) for the 4- th node, where N™ = [2, 3, 5, 6], the aggregation step is represented in formula (1) as For simplicity, it is omitted the window notation and it is referred to the graph convolution in formula (2) as X' = GraphConv(X), where X G RNxF= [xi, X2, . . . , XN ]Tis the set of node features.

[0099] GNNs typically include few graph convolutional layers due to the over-smoothing problem (see Jhony H Giraldo, Konstantinos Skianis, Thierry Bouwmans, and Fragkiskos D Malliaros, "On the trade-off between over-smoothing and over-squashing in deep graph neural networks", CIKM, 2023, reference [9]), where features tend to be more and more similar and thus less -SVT079-PC00- discriminative with the network depth. For this reason, it is employed an FFN to perform feature transformations and non-linear activations after the Window-based Grapher module 7 which has also a fully connected layer before and after the dynamic graph convolution layer as in reference

[0011] .

[0100] Besides, the graph representations are dynamically updated with / c-NN in every new layer as in reference

[0023] . Therefore, given an input feature X G RNxF, the overall Window-based Grapher module transfer function can be expressed as: where Win G RFx2Fand Wout £ R2FxFare learnable parameters of fully-connected layers that respectively increase and reduce the input feature dimension F, and e(-) is a nonlinear activation function. The bias terms and batch normalization are omitted in formula (3) for the sake of simplicity. The operation in formula (3) results in a new feature embedding Y G RNxF.

[0101] The FNN module 8

[0102] To encourage feature diversity, the output Y of the Window-based Grapher module 7 is processed by the FFN module 8 as follows. The FFN module 8 is implemented as a multi-layer perceptron with two fully connected layers with residual connection to Y given by:

[0103] Z = (7(YW1)W2f Y!(4) where Wi G RFxr projects the input features into an p-dimensional space, with p = F x E and E the hidden dimension ratio, and W2 G RrxFre-projects the features into the original F-dimensional space. The FFN module 8 in formula (4) also contains: (i) batch normalization layer after each linear projection Wi and W2, and (ii) bias terms that are omitted for the sake of simplicity.

[0104] Shi fted Windows

[0105] To introduce cross-window connections while maintaining the efficient computation described above, it is included in WiGNet a shifting operator similar to the one adopted in Swin Transfomer (see reference

[0024] ).

[0106] More precisely, it is implemented a Shifted Window-based Grapher module, where the graph construction and convolution are performed on shifted windows as -SVT079-PC00- illustrated in Figure 5.

[0107] Figure 5 shows an overview of the cycling operation used to obtain shifted windows. The top-left part of the feature maps is copied on the bottom-right part, then the masking mechanism is used to avoid connection between non-adjacent nodes in the original feature maps.

[0108] To do this, it is adopted a cycling operation to partition the feature map, and it is used a masking mechanism to allow connection only between nodes adjacent to the feature map.

[0109] In other words, multiple sub-graphs may arise in the same window as shown in Figure 3, where different colors and texture backgrounds are used to identify the masking mechanism (i.e., connections are allowed only between nodes that fall in the same color area). This phenomenon results in a heterogeneous construction of the graphs, implying a considerable drop in the number of neighboring nodes in certain regions. For instance, a node belonging to the top-left window of Figure 5 will be connected to k other nodes in that window out of Al x Al possible nodes, where M is the window size. Instead, a node in the section B of the bottom-right window will still be connected to k other nodes but out of SxS possible nodes, where S is the shiftsize typically set as 5 = . To attempt to solve this issue, the number of neighbors of each node is linearly adjusted by considering the maximum number of possible neighbors that the masking mechanism allows it to have. In particular, given k, the window-size Al x Al, and the number of possible neighbors for the node i (Pi), kL= p • k x can be used as the number of neighbors for that node.

[0110] Complexity considerations

[0111] Although both the method according to the present invention and ViG use k- NN to create the graph, one of the major advantages of WiGNet is the reduction in computational complexity as the image size increases. ViG's k-NN complexity, indeed, grows with the square of the number hw of nodes (patches) of the whole feature map and is given by: n C \ViCl • A>NN) ? - ( s.M / 2(5)

[0112] In contrast, the windowed approach of WiGNet results in a complexity that grows -SVT079-PC00- linearly with the number of patches as follows: where | Vw| is the number of nodes on each window w. Multiply- Accumulate (MACs) operations and memory footprint of ViG, WiGNet and two other graph-based models are compared in Figure 6.

[0113] Experiments

[0114] In this section, WiGNet is first experimented over the ImageNet-lK dataset (see Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al., "ImageNet large scale visual recognition challenge", IJCV, 115(3):211-252, 2015, reference", reference

[0033] ) comparing against ResNet (see reference

[0014] ), Pyramid Vision Tranformer (see Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao, "Pyramid vision transformer: A versatile backbone for dense prediction without convolutions", ICCV, 2021, reference

[0039] ), Swin Transformers (see reference

[0024] ), Poolformer (see Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan, "Metaformer is actually what you need for vision", CVPR, 2022, see reference

[0044] ), ViG (see reference

[0011] ), MobileViG (see reference

[0011] ), and GreedyViG (see Yan Han, Peihao Wang, Souvik Kundu, Ying Ding, and Zhangyang Wang. Vision HGNN, "An image is more than a graph of nodes", ICCV, 2023, see reference

[0012] ).

[0115] For the sake of comparability, k = 9 neighbors for graph construction as in ViG (see reference

[0011] ) are considered. Then, once the model according to the invention is trained on ImageNet, its adaptability is evaluated to a new classification task with higher-resolution images. To do this, it is used the tiny model as a pre-trained frozen backbone for facial identification on the

[0116] Celeb A-HQ dataset (see Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo, "Maskgan: Towards diverse and interactive facial image manipulation", CVPR, 2020, see reference

[0022] ). Finally, two ablation studies are performed on key design choices: whether or not to use shifting windows and which graph convolutional layer to -SVT079-PC00- adopt.

[0117] Experimental Setup

[0118] Datasets. In image classification, the benchmark dataset ImageNet ILSVRC 2012 (see reference

[0033] ) is commonly used as a standard evaluation metric. ImageNet contains approximately 1,2 million in training images and 50.000 in validation images, spanning across 1.000 categories.

[0119] The Celeb A-HQ dataset (see reference

[0022] ) is instead used to test the adaptability of the model according to the present invention in a downstream task with high-resolution images. Indeed, this dataset is a high-quality version of CelebA (see Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang, "Deep learning face attributes in the wild", ICCV, 2015, reference

[0025] ) that consists of 30.000 images. This dataset is used to perform a facial identification of 307 classes by rescaling the images to a resolution of 512 x 512. This rescaling is performed to be able to train the most memory-intensive models like ViG and GreedyViG.

[0120] Implementation details. For training all WiGNet models on ImageNet similar hyperparameters as ViG

[0011] are kept. It is adopted the commonly used training strategy proposed in DeiT (see reference

[0036] ) for fair comparison. The data augmentation includes RandAugment (see Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Rand, "Augment: Practical automated data augmentation with a reduced search space", CVPR Workshops, 2020, reference [4]), Mixup (see Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz, "Mixup: Beyond empirical risk minimization", ICLR, 2018, see reference

[0046] ), Cutmix (see Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo, "CutMix: Regularization strategy to train strong classifiers with localizable features", ICCV, 2019, see reference

[0045] ), random erasing (see Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang, "Random erasing data augmentation", AAAI, 2020, see reference

[0047] ). Additionally, for WiGNet-M it is adopted the repeated augmentation (see Elad Hoffer, Tai Ben- Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry, "Augment your batch: Improving generalization through instance repetition", CVPR, 2020, reference

[0015] ) and an Exponential Moving Average (EMA) scheme. The models according to -SVT079-PC00- the invention are implemented using PyTorch (see Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al., "Pytorch: An imperative style, high-performance deep learning library", NeurlPS, 2019, see reference

[0029] ) and all of them are trained on 8 GPUs NVIDIA GeForce RTX 3090.

[0121] Then, once the pre-trained models according to the invention are obtained, it is performed a transfer-learning experiment on higher-resolution images. In this context, all models are finetuned using Adam as an optimizer having a constant learning rate of 0,001 for 30 epochs using a Cross-Entropy loss function and a batch size of 64, except for GreedyViG and ViG where it was used a batch-size of 16 for memory reasons.

[0122] Main results

[0123] First, the classification results on ImageNet are provided. Then, it is shown that the pre-trained backbone according to the invention achieves a better trade-off between accuracy and complexity than other models using CelebA-HQ as a downstream task.

[0124] ImageNet. Figure 7 shows the comparison between the method according to the present invention and previous state-of-the-art methods. WiGNet outperforms or achieves competitive results against previous state-of-the-art models for similar complexity. For instance, comparing WiGNet with non-graph-based models, the tiny model according to the present invention with 78,8 of accuracy outperforms all previous methods for low MAGs (around 2G), and a small number of parameters (around 10M). Similarly, the WiGNet-S achieves better results than the Swin-T model with comparable MAGs and than ResNet-152 with almost half the parameters.

[0125] In addition, WiGNet shows competitive results against the previous graphbased method under similar conditions. Moreover WiGNet-Ti trained with slightly larger images (256 x 256) and using a window size of 8x8, works better than the same model trained on (224 x 224) images since in this case the window-size is smaller (7 x 7) and thus the number of possible neighbors.

[0126] The ResNet are taken also from Ross Wightman, Hugo Touvron, and Herve -SVT079-PC00-

[0127] Jegou, "ResNet strikes back: An improved training procedure in timm", NeurlPS, 2021, reference

[0042] ,

[0128] CelebA-HQ. To show the adaptability of WiGNet to new classification tasks having higher resolution images, experiments are conducted using the pre-trained model according to the invention on ImageNet as a frozen backbone on CelebA-HQ dataset (see reference

[0022] ) as a downstream facial identity classification task. Particularly, a new classification layer was trained on these features keeping the rest of the architecture frozen.

[0129] Figure 8 shows the results obtained by the backbone according to the present invention compared to other graph-based models in terms of accuracy, memory usage, and MACs using 512 x 512 resolution images. In this context it is noticed that ViG struggles to converge, while the backbone according to the invention achieves the second-best result, only outperformed by GreedyViG. However, by comparing the memory footprint required for each model, it is observed that WiGNet needs only 0,5 GB, while for GreedyViG the occupancy is ~ 3x more. MobileViG, instead, is the model with the lowest MACs. Nevertheless, it occupies more memory than WiGNet achieving worse results.

[0130] It is clear from Figure 8 that WiGNet is the closest model to the optimal point (i.e. top-left corner of the plot), achieving similar Top-1 accuracy results to GreedyViG but using significantly less memory, even compared to MobileViG. In Figure 6 it is also analyzed the complexity of these models in terms of MACs and memory as the resolution of the input image increases. From these results it is observed that WiGNet computation and memory requirements scale only linearly with the image size. By comparison, the memory requirements for GreedyViG (and for ViG also the complexity in terms of MACs) scales quadratically with the image size. These results show that WiGNet can operate with images of high resolution using less memory than MobileViG while maintaining the complexity under control.

[0131] Ablation Studies

[0132] Shifted windows. Figure 9 shows the results when the WiGNet uses the Shifted Window-based Grapher module previously explained. It is observed that contrary to the Swin Transformer, the shifting strategy does not bring any advantage to WiGNet -SVT079-PC00- in this context, despite the results seeming to improve slightly by increasing the model size. It is hypothesized that, because of the low resolution of the images in ImageNet, is possible to independently analyze the windows and still obtain good results. Therefore, it is conducted the same transfer learning experiment described in the "Main Result" section to monitor the behavior of the backbone according to the present invention without shifting and with higher-resolution images, ablating also on the Adaptive k-NN strategy. In Figure 10 it is observed that for larger images the shifting operator is crucial, allowing for a gain of almost 2% points on average, and a significantly lower standard deviation. Moreover, this gain increases to 6% when the Adaptive k-NN strategy is implemented.

[0133] Graph Convolutional Operator. Finally, it is conducted an ablation study with some well-known graph convolutional functions in the Window-based Grapher module 7, including Max-Relative GraphConv (see reference

[0023] ), GraphSAGE (see William L Hamilton, Rex Ying, and Jure Leskovec, "Inductive representation learning on large graphs", NeurlPS, 2017, see reference

[0010] ) and EdgeConv (see Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon, "Dynamic graph CNN for learning on point clouds", ACM TOG, 38(5):1- 12, 2019, see reference

[0041] ).

[0134] Figure 11 shows the results of this experiment when the WiGNet-Ti model is trained on ImageNet without the shifting operator, as it seems to work slightly better for the tiny size model. It is observed that the Max-Relative graph convolution achieves competitive results with less complexity than the other operators.

[0135] The present description has tackled some of the possible variants, but it will be apparent to the man skilled in the art that other embodiments may also be implemented, wherein some elements may be replaced with other technically equivalent elements. The present invention is not therefore limited to the explanatory examples described herein, but may be subject to many modifications, improvements or replacements of equivalents parts and elements without departing from the basic inventive idea, as set out in the following claims.

Claims

-SVT079-PC00-CLAIMS1. A method for processing an image using a vision graph neural network, said vision graph neural network comprising a window-based grapher module (7) including a first fully connected layer with batch normalization (9), a windows partitioning module (10), a dynamic graph convolution module (11), a windows reverse module (12), a second fully connected layer with batch normalization (13) and a skip connection (15), wherein said window-based grapher module (7) is configured to: process a feature vector (X) of said image (2) through said first fully connected layer with batch normalization (9) to obtain a normalized feature vector; partition said normalized feature vector into a plurality of non-overlapping windows using said windows partitioning module (10); for each window, construct a graph where nodes represent patches of said image (2) within the respective window and edges represent relationships between said nodes, and apply a graph convolutional operation to each graph to update node features within each window using said dynamic graph convolution module (11); reshape the updated node features from each window back into the format of said normalized feature vector using said windows reverse module (12); process the reshaped feature vector through said second fully connected layer with batch normalization (13); combine said feature vector (X) directly with an output of said second fully connected layer with batch normalization (13) using said skip connection (15).

2. A method for processing an image using a vision graph neural network according to claim 1, wherein said graph convolutional operation applied in said dynamic graph convolution module (11) comprises using a k-Nearest Neighbors algorithm to determine said edges between said nodes based on feature similarity.

3. A method for processing an image using a vision graph neural network according to claim 2, wherein said k-Nearest Neighbors algorithm dynamically adjusts a number of neighbors considered for each node based on the position of the node within said respective window.

4. A method for processing an image using a vision graph neural network-SVT079-PC00- according to claim 1, wherein said graph convolutional operation comprises a Max- Relative graph convolution, which aggregates features from neighboring nodes and updates said node features using a learnable weight matrix.

5. A method for processing an image using a vision graph neural network according to claim 1, wherein said windows partitioning module (10) is configured to partition said normalized feature vector into windows of a fixed size.

6. A method for processing an image using a vision graph neural network according to claim 1, further comprising aggregating said combined feature vector (X) and said output of said second fully connected layer with batch normalization (13) to form a global feature representation of said image (2).

7. A method for processing an image using a vision graph neural network according to claim 6, further comprising classifying said image (2) based on said global feature representation of said image (2).

8. A method for processing an image using a vision graph neural network according to claim 1, further comprising applying a shifting operation to said plurality of non-overlapping windows in said windows partitioning module (10) to create shifted windows.

9. A method for processing an image using a vision graph neural network according to claim 8, wherein the shifting operation involves: cyclically shifting portions of the normalized feature vector to ensure that each window overlaps with adjacent windows; and using a masking mechanism to control the connections between nodes in the shifted windows to ensure that only adjacent nodes are connected.

10. A method for processing an image using a vision graph neural network according to claim 9 or 10, wherein the cyclically shifting operation involves copying the top-left part of the normalized feature vector to the bottom-right part to create the shifted windows.

11. A method for processing an image using a vision graph neural network according to claim 8, wherein said dynamic graph convolution module (11) recalculates the graph structure and updates the node features based on the shifted windows to capture cross-window connections.-SVT079-PC00-12. A device (4) for processing an image using graph-based neural network operations, comprising: a first fully connected layer with batch normalization (9) configured to receive a feature vector (X) of said image (2) and to process it to obtain a normalized feature vector; a windows partitioning module (10) configured to partition said normalized feature vector into a plurality of non-overlapping windows; a dynamic graph convolution module (11) configured to, for each window, construct a graph where nodes represent patches of said image (2) within the respective window and edges represent relationships between said nodes, and apply a graph convolutional operation to each graph to update node features within each window; a windows reverse module (12) configured to reshape the updated node features from each window back into the format of said normalized feature vector; a second fully connected layer with batch normalization (13) configured to process the reshaped feature vector; a skip connection (15) configured to combine said feature vector (X) directly with an output of said second fully connected layer with batch normalization (13).

13. A modular block (4) comprising said device for processing an image using graphbased neural network operations (7) according to claim 12 and a Feed Forward Network module (8).

14. A vision graph neural network comprising:- a stem block (3) for extracting features from an image (2);- at least one modular block (4) according to claim 13 followed by a respective downsampling block (5);- a fully connected layer (6) for producing class scores and classifying said image (2).

Images