Structured data representation method based on semantic space two-channel composition

Abstract

The invention relates to a double-graph neural network framework, namely a Vision DualGNN (VDG), for a visual task. Aiming at the problems of scattered object neighbor distribution and insufficient graph processing caused by the fact that an existing Vision GNN (ViG) method only depends on semantic information when a graph structure is constructed, the VDG is used for respectively processing graphs constructed based on space and semantics by fusing space and semantic information and adopting a double-flow graph neural network architecture, so that the graph structure is constructed based on the space and the semantics. Therefore, the expression ability of the node features in the image and the expression effect of the graph level are effectively improved. Specifically, the method comprises the following steps: 1) converting an image block into a node feature with high expressivity by using a node encoder; (2) a double-graph structure is constructed, a spatial graph serves as a constraint condition of a semantic graph, and the spatial perception ability of node features is enhanced; experimental results show that the VDG model provided by the invention has better performance than other baseline models on ImageNet and CIFAR-100 data sets, and the effectiveness and superiority of the VDG model in a visual task are proved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to a structured data representation method based on semantic space dual-channel composition. The method is applicable to image representation methods in the field of computer vision. Background Art

[0002] Deep learning belongs to the category of artificial intelligence and machine learning. It constructs neural network structures and extracts features to achieve the expression of structured data such as images. In recent years, deep learning has achieved good expression capabilities on images. Models based on convolutional neural networks, transformers, and graph neural networks have made significant progress in the field of image representation after training on large-scale subjective databases, such as EP4302256 (A1). The background of this method is based on the great potential of graph neural networks (GNNs) in visual tasks, but it also faces the challenge of effectively building and processing graph structures. In the field of visual tasks, graph neural networks (GNNs) have attracted widespread attention due to their ability to process non-Euclidean data. However, traditional methods have encountered some difficulties in building and processing graph structures, especially in how to effectively utilize spatial and semantic information in images. Although the existing Vision GNN (ViG) method can construct a graph by dividing the image into blocks and treating them as nodes, and then connecting them using the semantic similarity between nodes, this often leads to the decentralization of node distribution, fails to fully utilize the spatial information of the image, and limits the performance of the model in understanding the image content. Therefore, how to effectively integrate spatial information while maintaining semantic information has become one of the key challenges in improving the performance of graph neural networks in visual tasks. Summary of the invention

[0003] The present invention aims to solve the problem that the existing vision graph neural network (Vision GNN, ViG) relies only on semantic information when building the graph structure, resulting in scattered node distribution and poor graph processing effect. To this end, the present invention proposes a vision dual graph neural network (Vision DualGNN, VDG) architecture, which optimizes the construction and processing of the graph structure by simultaneously utilizing the spatial and semantic information of the image to improve the performance of visual tasks. The specific implementation method is as follows: S1, divide the input image into multiple patches according to the predetermined size H×W×3, each patch is regarded as a node Xi, where i=1,2,...,N, N is the total number of nodes. Each node Xi has a fixed feature dimension C; S2, apply the node encoder to encode each node Xi to generate a richer node feature representation. The node encoder combines channel attention and spatial attention mechanisms through multi-layer convolution operations to enhance the expressiveness of node features and ensure that each node can more accurately reflect the corresponding image area information. ; S3, constructs two graph structures for the generated graph nodes, using the physical space graph to make up for the problem that the semantic feature graph is insufficient in describing the position encoding; S4, uses the designed dual-stream graph neural network (DualGNN) to process the semantic graph G1 and the spatial graph G2 respectively. DualGNN consists of two parallel GNN layers, each of which is responsible for processing the corresponding graph structure. Through this design, the model can simultaneously consider the spatial and semantic information of the node during the processing process, further improving the expressiveness of the graph structure and the generalization ability of the model; S5, after completing the dual-stream graph neural network processing, the node features from the semantic graph and the spatial graph are fused to obtain the final node feature representation. These features are then fed into the classifier or other task-specific modules to complete the final visual task prediction; Furthermore, in step S3, specifically: S31, first, in order to construct a semantic graph, we need to calculate the semantic distance between each node and all other nodes. The semantic distance used here refers to the similarity measure between node feature vectors, which reflects the relevance of nodes in image content; S32, feature extraction: For each node , we have obtained its feature representation through the node encoder. Assume that the node The characteristic is expressed as ,but is a vector of length C, where C is the feature dimension; S33, Calculate semantic distance: Next, we need to calculate the distance between any two nodes and The semantic distance s(vi, vj) between them. This process can be implemented in many ways, such as using cosine similarity, Euclidean distance, etc. In the present invention, we recommend using cosine similarity because it can better reflect the consistency of the directions of two vectors and is more suitable for measuring the semantic similarity between feature vectors. The calculation formula of cosine similarity is: in, Representation vector and The dot product of and Represents vectors and The norm of S34, select neighbor nodes: After calculating the semantic distances between all node pairs, for each node , we select the K nodes with the smallest semantic distance as its neighbor nodes. Here, K is a hyperparameter that can be adjusted according to the specific task requirements and dataset characteristics. The process of selecting neighbor nodes can be formalized as finding the nodes corresponding to the first K minimum semantic distances of each node; S35, Constructing a semantic graph: Once the neighbor nodes of each node are determined, the semantic graph can be constructed by adding edges. }, V represents a node set, and represents the edge set selected based on semantic distance. For each node ,if is one of its selected neighbor nodes, then and Add an edge between , indicating that the two nodes have a strong semantic association; S36,Similar to the semantic graph, the construction of the spatial graph is also based on the relationship between nodes, but this time the focus is on the physical position relationship of the nodes in the image; S37, Position encoding: In order to calculate the spatial distance, we first need to assign a position encoding to each node. This position encoding can be simply based on the original position of the node in the image, that is, the center coordinates of the image block where the node is located. Assume that the node The position of is encoded as pi, then pi can be represented as a two-dimensional vector ( , ),in and The nodes are The horizontal and vertical coordinates of the center point of the block; S38, calculate the spatial distance: Next, calculate the distance between any two nodes and The spatial distance between two points is d(vi,vj). Euclidean distance is used here because it directly reflects the straight-line distance between two points in space. The calculation formula for spatial distance is: ; S39, select neighbor nodes: Similar to the construction process of semantic graph, for each node , we select the L nodes with the closest spatial distance to it as its neighbor nodes. L is also a hyperparameter and can be adjusted according to actual conditions. The process of selecting neighbor nodes can be formalized as finding the nodes corresponding to the first L minimum spatial distances of each node. After determining the neighbor nodes of each node, we construct a spatial graph by adding edges. In the spatial graph G2={V, }, V represents a node set, and Represents a set of edges selected based on spatial distance. For each node ,if is one of its selected neighbor nodes, then and Add an edge between , indicating that the two nodes are adjacent in space. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0005] Figure 1 is an overall framework diagram of a structured data representation method based on a semantic space dual-channel composition according to an embodiment of the present invention, Figure 2 is the corresponding node encoder part representation. DETAILED DESCRIPTION

[0006] In an embodiment of the present invention, we will describe in detail a structured data representation method based on a semantic space dual-channel composition. It should be noted that the technical solution discussed here is only a partial example of the present invention, not the entire content. Any other examples implemented by ordinary technicians based on these examples without creative work will fall within the protection scope of the present invention.

[0007] This example will be described based on the ImageNet dataset and CIFAR-100. ImageNet is a large-scale visual dataset containing more than 14 million images covering a variety of categories, and is often used for image classification and object recognition tasks for training and evaluating deep learning models. CIFAR-100 is a small-scale image classification dataset containing 60,000 32x32 color images in 100 categories, and is often used for fast experiments and benchmarking of machine learning and deep learning models.

[0008] First, we conduct preliminary experiments on the proposed two-stream GNN architecture based on Vision GNN on the CIFAR-100 dataset. This architecture combines semantic graphs with spatial graphs for better feature extraction. After verifying the effectiveness of the architecture through CIFAR-100 experiments, we further conduct experiments on the ImageNet dataset to evaluate the generalization performance of the new architecture.

[0009] The embodiment of the present invention provides a structured data representation method based on a semantic space dual-channel composition, such as Figure 1 and Figure 2 As shown, the specific steps are as follows: S1, given an input image of size H×W×3, first normalize it to ensure that the color channel value range of the image is between 0 and 1. The input image is evenly divided into multiple patches of equal size, and each patch is regarded as a node Xi; S2, extract the initial feature vector of each node Xi through a pre-trained convolutional neural network (CNN) This CNN can be any suitable model such as ResNet, VGG, etc. For each node’s feature vector The channel attention mechanism is applied to enhance the expressiveness of features. Based on the channel attention, the spatial attention mechanism is further applied to enhance the spatial correlation of node features. S31, for each node Xi, calculate the semantic distance s(vi, vj) between it and all other nodes Xj, use cosine similarity as the distance metric, and select K semantically nearest neighbor nodes for each node Xi based on the calculated semantic distance. K is a hyperparameter that can be adjusted according to task requirements. Based on the selected neighbor nodes, construct a semantic graph G1={V, }, where V represents a node set, represents the set of edges selected based on semantic distance; S32, for each node Xi, calculate the spatial distance d(vi, vj) between it and all other nodes Xj, using Euclidean distance as the distance metric. Based on the calculated spatial distance, select L spatial nearest neighbor nodes for each node Xi. L is a hyperparameter that can be adjusted according to task requirements. Based on the selected neighbor nodes, construct a spatial graph G2={V, }, where V represents a node set, represents the edge set selected based on spatial distance; S4, uses two parallel graph neural network (GNN) layers to perform graph convolution operations on the semantic graph G1 and the spatial graph G2 respectively. Each GNN layer updates point features through a message passing mechanism, gradually enhancing the expressiveness of node features. Based on the output of each GNN layer, a feature fusion module is used to fuse the node features of the semantic graph and the spatial graph to generate a comprehensive feature vector. Feature fusion can be achieved by simple weighted summation, concatenation, and then passing through a fully connected layer; S5, after being processed by the dual-stream graph neural network, each node obtains a new feature representation. These feature representations are concatenated to form the final image feature vector. The final image feature vector is sent to a classifier for training. The classifier can be a fully connected layer or a more complex network structure for mapping features to category labels. The performance of the model is evaluated on standard datasets (such as ImageNet and CIFAR100), and the effectiveness of the present invention is verified by comparing the classification accuracy of different models. Dataset Introduction

[0010] CIFAR-100 (Canadian Institute For Advanced Research) is an extended version of CIFAR-10, created by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton of the University of Toronto. It contains a variety of object categories in daily life and is a representative of small image datasets in machine learning and computer vision. It contains 60,000 color images with a size of 32x32 pixels, divided into 50,000 training images and 10,000 test images.

[0011] ImageNet is one of the largest and most influential image datasets. It was created by Professor Fei-Fei Li's team at Stanford University and has promoted the development of deep learning through the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). It contains more than 14 million images, covering a variety of real-world objects. The commonly used subset of ILSVRC is ImageNet-1K, which contains 1.2 million training images, 50,000 verification images, and 100,000 test images. Experimental analysis

[0012] In order to verify the effectiveness and superiority of the Vision DualGNN (VDG) proposed in this paper, we conducted extensive experiments on multiple standard datasets and compared them with existing mainstream vision models. The following is a detailed experimental analysis.

[0013] We conducted experiments on the following two standard datasets: ImageNet ILSVRC 2012: contains about 1.28 million training images and 50,000 validation images, with a total of 1,000 categories; CIFAR-100: Contains 60,000 small 32×32 images of 100 categories, divided into 50,000 training images and 10,000 test images.

[0014] We design three VDG model variants of different scales: VDG-Ti: 12 layers deep, 192 feature dimensions, about 9.1M parameters, and about 1.8B FLOPs; VDG-S: 16 layers deep, 320 feature dimensions, about 29.4M parameters, and about 6.5B FLOPs; VDG-B: 16 layers deep, 480 feature dimensions, about 64.5M parameters, and about 14.4B FLOPs.

[0015] Hyperparameter settings during model training: Optimizer: Adam optimizer, initial learning rate is 0.001, minimum learning rate is 1e-6; Batch size: 1024 on ImageNet and 256 on CIFAR-100; Training rounds: 300 epochs on ImageNet and 300 epochs on CIFAR-100; Hardware environment: All models are trained on 8 NVIDIA A100 GPUs.

[0016] Table 1. Comparison of the model's performance on the ImageNet dataset with other models

[0017]

[0018] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A structured data representation method based on semantic space dual-channel composition, characterized in that: The following steps are involved: S1, image preprocessing: divide the input image into multiple blocks, each block as a node; S2, node encoding: Apply the node encoder module to encode each node and generate enhanced node feature representation; S3, build a dual graph structure: S3.1, semantic graph construction: Calculate the semantic distance between each node and other nodes, select the K semantic nearest neighbor nodes of each node, and construct the semantic graph. The calculation method of the centralized semantic distance is as follows: in, Representation vector and The dot product of and Represents vectors and The norm of ; S3.2, spatial graph construction: Calculate the spatial distance between each node and other nodes, select the L spatial nearest neighbor nodes of each node, and construct a spatial graph, where the spatial distance is calculated as follows: in Represents the node The spatial distance , Represents nodes location information; S4, dual-stream graph neural network processing: use the dual-stream graph neural network module to process the semantic graph and spatial graph respectively to generate a comprehensive feature vector; S5, feature fusion and output: send the comprehensive feature vector into the classifier to achieve image classification task.

2. The structured data representation method based on semantic space dual-channel composition according to claim 1 is characterized in that: The S2 includes: S2, a node encoder module is proposed to encode each node of the input image and generate enhanced node feature representation; this module mainly uses a channel attention mechanism to calculate the query vector, key vector and value vector of each node feature vector, and updates the node features through the dot product attention mechanism. The spatial attention mechanism is used to generate spatial attention weights and re-weight the node features to further enhance the spatial correlation of the node features.

3. The structured data representation method based on semantic space dual-channel composition according to claim 1 is characterized in that: S3 The semantic graph construction unit uses cosine similarity as a measure of semantic distance, and the spatial graph construction unit uses Euclidean distance as a measure of spatial distance.

4. The structured data representation method based on semantic space dual-channel composition according to claim 1 is characterized in that: The system provides three model variants of different sizes, and the specific parameter settings are as follows: VDG-Ti: 12 layers in depth, 192 feature dimensions, approximately 9.1M parameters, and approximately 1.8B FLOPs; VDG-S: 16 layers in depth, 320 feature dimensions, approximately 29.4M parameters, and approximately 6.5B FLOPs; VDG-B: 16 layers in depth, 480 feature dimensions, approximately 64.5M parameters, and approximately 14.4B FLOPs.

5. The structured data representation method based on semantic space dual-channel composition according to claim 1 is characterized in that: The dual-stream graph neural network module uses two parallel graph neural network layers to process the semantic graph and the spatial graph respectively; and fuses the node features of the semantic graph and the spatial graph through the feature fusion module to generate a comprehensive feature vector.

6. The structured data representation method based on semantic space dual-channel composition according to claim 1, characterized in that: The dataset contains 1.2 million images from ImageNet and 60,000 images from CIFAR-100.

Images