An image classification method based on a graph neural network

Abstract

The application provides a kind of image classification method and device based on graph neural network, electronic equipment and storage medium, relating to image processing technical field, solve the graph learning in the subsequent graph module of convolutional neural network, lack the key information of graph learning in the graph neural network generation graph learning process based on edge label, leading to the keyness and completeness of the generated picture feature information is poor, the classification accuracy of image is low.The method comprises: configuring sample image classification task;Configure sample image public data set;Set the learning parameters of sample image public data set;Based on neural network model, calculate the classification result of sample image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image classification method, apparatus, electronic device, and storage medium based on graph neural networks. Background Technology

[0002] Image classification is a crucial area in computer vision. Optimizing neural network model parameters using large-scale labeled samples can significantly improve classification accuracy. However, due to privacy concerns, security concerns, and the high cost of labeled data, many applications in fields such as medicine, satellite technology, and the military struggle to obtain sufficient labeled training samples. Currently, convolutional neural networks (CNNs) or edge-label-based graph neural networks (GNNs) are commonly used to address this issue. However, CNNs can only extract features with fixed receptive fields, resulting in shallow details and limited feature information. This hinders subsequent GNN modules from effectively learning and updating the graph using node features, ultimately impacting classification accuracy. While edge-label-based GNNs update edges based on intra-class similarity and inter-class dissimilarity and classify query set nodes using edge labels, they neglect node learning and updating.

[0003] Therefore, a new technological approach is urgently needed to solve the above problems. Summary of the Invention

[0004] This application provides an image classification method, apparatus, electronic device, and storage medium based on graph neural networks, which solves the problems of lack of key information in the graph learning of subsequent graph modules in convolutional neural networks and poor criticality and completeness of generated image feature information due to the lack of good updating of node information during the graph learning process of graph neural networks based on edge labels, resulting in low image classification accuracy.

[0005] In a first aspect, this application provides an image classification method based on graph neural networks, the method comprising: Configure sample image classification tasks; each classification task includes a support set and a query set. The support set includes N classes of sample images, and each class of sample images includes K label samples. Configure a public dataset of sample images; the public dataset includes mini-ImageNet and tiered-ImageNet; mini-ImageNet includes 100 class images, each class includes 600 84×84 pixel RGB color images; tiered-ImageNet includes 34 superclass images, each superclass includes a first preset interval of subclass images; Set the learning parameters for the public dataset of sample images; the learning parameters include the initial learning rate, weight decay coefficient, mini-batch size for meta-training tasks, and learning rate decay value; The classification results of sample images are calculated based on a neural network model. The neural network model includes a feature extraction module and a side label graph neural network module. The feature extraction module is used by the neural network branches and attention branches to extract and fuse the local and global features of the input sample images. The side label graph neural network module is used for learning and updating node feature information and side label feature information.

[0006] In one possible implementation, the classification result of the sample image is calculated based on a neural network model, including: The features of the sample image are calculated and obtained based on the feature extraction module; An initial connected graph is established, and the similarity values ​​between nodes in the initial connected graph are calculated based on the edge label graph neural network module to obtain the updated connected graph. Based on the updated connected graph, the similarity value between the set node and other nodes is calculated using the Softmax function to obtain the classification result of the sample image.

[0007] In one possible implementation, the feature extraction module includes three parallel modules and a fully connected layer. Each parallel module includes three two-dimensional convolutional layers, a spatial attention mechanism, a channel attention mechanism, five batch normalization layers, two activation functions, and a max pooling layer. The channel attention mechanism includes a compression network and an activation network. The step of calculating and obtaining features of the sample image based on the feature extraction module includes: The feature map F1 of the input image is obtained by calculating the two-dimensional convolutional layer and max pooling. Feature map F2 is obtained by performing convolution calculation based on feature map F1; Based on feature map F2, different feature information is calculated and obtained through the feature extraction module, and the different feature information is summed to generate feature map F3; The number of channels in the feature extraction module is changed to the second and third channels respectively. Based on the feature map F2, different feature information is calculated and obtained by the feature extraction module. The different feature information is summed to generate feature map F4. Features of sample images are calculated using a fully connected network structure.

[0008] In one possible implementation, different feature information is calculated and obtained based on feature map F2 through a feature extraction module, and the different feature information is summed to generate feature map F3, including: Based on feature map F2, convolution calculations are performed at the first and second scales respectively to obtain the feature information F3 at the first scale. 1 Second-scale feature information F3 1 ; The spatial attention mechanism module performs weighted calculations on each pixel in feature map F2 to obtain important region feature information F3. 2 ; Feature information F3 is obtained by computing based on compressed networks and excitation networks. 3 ; The first-scale feature information F3 1 Second-scale feature information F3 1 Important regional feature information F3 2 and feature information F3 3 Summation is performed, and the summation result is used to calculate the feature map F3 based on the activation function and max pooling.

[0009] In one possible implementation, an initial connected graph is established, and the similarity values ​​between nodes in the initial connected graph are calculated based on the edge-labeled graph neural network module to obtain an updated connected graph, including: An initial connected graph G is constructed based on the support set of the input sample images and the features of the query sample images; The initial fully connected graph G is calculated based on the edge-labeled graph neural network module, and the updated connected graph is obtained.

[0010] In one possible implementation, calculating the initial fully connected graph G based on the edge-labeled graph neural network module and obtaining the updated connected graph includes: The node update network is used to calculate the node features and edge features in the initial fully connected graph G to obtain the updated node features; the node update network includes aggregation network, connection network and multilayer sensor; Calculate updated edge label features based on the updated node feature information; Calculate the attention coefficient between node q in the query set and its neighbor node p; The attention coefficient is normalized using the Softmax function; The updated query set node feature information is obtained by calculating the attention coefficients of the query set nodes and their neighbor nodes after normalization using a weighted summation.

[0011] In one possible implementation, based on the updated connected graph, the similarity value between set nodes and other nodes is calculated using the Softmax function to obtain the classification result of the sample image, including: After updating the node and edge label feature information based on the three-layer edge label graph neural network module, the final node features and final edge label features are calculated and obtained. The classification results of the sample images are calculated based on the Softmax function.

[0012] Secondly, this application provides an image classification device based on a graph neural network, comprising: Configuration unit, setting unit, calculation unit.

[0013] The configuration unit is used to configure the sample image classification task; each classification task includes a support set and a query set, the support set includes N classes of sample images, and each class of sample images includes K label samples; The configuration unit is also used to configure the public dataset of sample images; the public dataset includes mini-ImageNet and tiered-ImageNet; mini-ImageNet includes 100 class images, each class includes 600 84×84 pixel RGB color images; tiered-ImageNet includes 34 superclass images, each superclass includes a first preset interval of subclass images; The setting unit is used to set the learning parameters of the public dataset of sample images; the learning parameters include the initial learning rate, weight decay coefficient, mini-batch value of the meta-training task, and learning rate decay value; The computing unit is used to calculate the classification results of sample images based on a neural network model. The neural network model includes a feature extraction module and a side label graph neural network module. The feature extraction module is used to extract and fuse local and global features of the input sample image by neural network branches and attention branches. The side label graph neural network module is used to learn and update node feature information and side label feature information.

[0014] Thirdly, an electronic device is provided, comprising: a processor and a memory; wherein the memory is used to store one or more programs, the one or more programs including computer execution instructions, and when the electronic device is running, the processor executes the computer execution instructions stored in the memory to cause the electronic device to perform an image classification method based on a graph neural network as described in the first aspect.

[0015] Fourthly, a computer-readable storage medium is provided for storing one or more programs, the one or more programs including instructions that, when executed by a computer, cause the computer to perform an image classification method based on a graph neural network as described in the first aspect.

[0016] It should be noted that the technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of the corresponding implementation methods in the first aspect, and will not be repeated here.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0018] Figure 1 A schematic diagram of an electronic device structure provided for an embodiment of this application; Figure 2A schematic flowchart of an image classification method based on a graph neural network is provided for an embodiment of this application; Figure 3 A schematic flowchart of another image classification method based on graph neural networks provided for embodiments of this application; Figure 4 A schematic flowchart of another image classification method based on graph neural networks provided for embodiments of this application; Figure 5 A schematic flowchart of a specific image classification method based on a graph neural network is provided for the embodiments of this application; Figure 6 A schematic diagram illustrating yet another specific image classification method based on graph neural networks provided for embodiments of this application; Figure 7 A schematic diagram illustrating another specific image classification method based on graph neural networks provided for embodiments of this application; Figure 8 A schematic flowchart of another image classification method based on graph neural networks is provided for the embodiments of this application; Figure 9 A schematic diagram of an image classification method device based on a graph neural network provided for embodiments of this application; Figure 10 An example image of a sample dataset provided for embodiments of this application; Figure 11 A schematic diagram of an image classification process implemented by a neural network is provided for an embodiment of this application; Figure 12 A flowchart illustrating a channel attention mechanism (left) and a spatial attention mechanism (right) provided for embodiments of this application; Detailed Implementation The following describes in detail, with reference to the accompanying drawings, an image classification method and apparatus based on graph neural networks provided in this application. The term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The terms "first" and "second," etc., in the specification and drawings of this application are used to distinguish different objects or different processing of the same object, not to describe a specific order of objects. Furthermore, the terms "comprising" and "having," and any variations thereof, mentioned in the description of this application, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0019] This application provides an image classification method based on graph neural networks. In this method, the feature extraction module combines neural network branches and attention branches in parallel to extract and effectively fuse local and global features of the input image. The network then extracts deep and key information as node features. The edge-label graph neural network based on the node attention mechanism alternately updates the feature information of nodes and edges through intra-class similarity, inter-class dissimilarity, and direct connections between layers, achieving learning and updating of nodes and edges. Finally, the classification results of small sample images are output through the Softmax function. Figure 2 As shown, this application provides an image classification method based on a graph neural network, which includes steps S201-S204.

[0020] S201. Configure the sample image classification task.

[0021] Few-shot classification is the process of learning a classifier when there are only a few training samples for each data class. Each few-shot classification task T consists of two parts: a support set S (i.e., a set of input samples with labels, similar to a training set) and a query set Q (i.e., a set of unlabeled samples, used to evaluate the learned classifier, similar to a test set). When the support set S contains N classes of samples, and each class has K labeled samples, the problem is called an N-way K-shot classification problem.

[0022] The training and testing of the few-shot classification task employs an episode-based training mechanism. This task can be represented by the formula: T = S∪Q, where... and , where T represents the number of query samples, xi and yi∈{A1,…AN}=AN⊂A represents the i-th input data and its corresponding label; the dataset used for training or testing contains a set of categories called A, where Atrain∩Atest=∅.

[0023] S202, Configure the public dataset for sample images.

[0024] As shown in Table 1, this invention uses two public datasets, mini-ImageNet and tiered-ImageNet, for training, validation, and testing. mini-ImageNet is a small-sample dataset proposed in Matching networks, extracted from the original ILSVRC-2012 dataset. All datasets consist of RGB color images, 84×84 pixels in size, with 100 different categories extracted, each containing 600 samples. This invention uses the dataset in a 64:16:20 ratio for training, validation, and testing, ensuring no overlap among the three datasets. Tiered-ImageNet is similar to mini-ImageNet, also a subset extracted from ILSVRC-2012. It extracts 34 superclasses, each containing 10-30 subclasses, for a total of 608 subclasses. Each subclass contains a varying number of images, totaling 779,165 images. Unlike the mini-ImageNet dataset, the tiered-ImageNet dataset is much larger. In this invention, the class hierarchy of ImageNet was considered in the dataset partitioning. The extracted superclasses were divided into 20 superclasses (351 subclasses) as the training set, 6 superclasses (97 subclasses) as the validation set, and 8 superclasses (160 subclasses) as the test set, which is a ratio of 20:6:8 for training, validation and testing. The average number of images in each class is 1281, while ensuring that there is no overlap between the three datasets.

[0025] Table 1 S203. Set the learning parameters for the public dataset of sample images.

[0026] This invention employs a 4-way 1-shot few-shot learning setup for the dataset in step S202. This experiment is implemented in PyTorch and trained using the Adam optimizer. In one embodiment, the initial learning rate is set to 125 × 10⁻⁶. -4 The weight decay factor is 10. -6 In the experiment, the mini-batch size for the meta-training task was set to 40. The mini-ImageNet dataset was used, and the learning rate decayed by 0.1 after every 15,000 iterations. Since tiered-ImageNet is a larger dataset than mini-ImageNet and requires more iterations to converge, the learning rate was decayed by 0.1 after every 30,000 iterations.

[0027] S204. Calculate the classification results of the sample images based on the neural network model.

[0028] The neural network model consists of two parts: a feature extraction module and a node attention-based edge-labeled graph neural network (N&E GNN) module. It should be noted that, in one embodiment, the specific structure is as follows... Figure 3 As shown. It should be noted that, in one embodiment, the feature extraction module can be a parallel module of a dual-branch network and an attention mechanism. The parallel module of a dual-branch network and an attention mechanism combines the neural network branches and the attention branch in parallel to achieve the extraction and effective fusion of local and global features of the input image by the two branches. The network then extracts deep and key information as node features. The edge label graph neural network based on the node attention mechanism alternately updates the feature information of nodes and edges through intra-class similarity, inter-class dissimilarity, and direct connections between layers, realizing the learning and updating of nodes and edges. Finally, the classification results of small sample images are output through the Softmax function.

[0029] In one possible implementation, such as Figure 3 The specific image classification method based on graph neural networks shown may include steps S2041-S2043 in step S2044.

[0030] S2041. Calculate and obtain the features of the sample image based on the feature extraction module.

[0031] Features are extracted from the input small sample images through a parallel module of dual-branch network and attention mechanism.

[0032] In one possible implementation, such as Figure 4 The image classification method based on graph neural networks shown may include steps S2041A-S2041E in particular.

[0033] S2041A: The feature map F1 of the input image is obtained by calculating based on two-dimensional convolutional layers and max pooling.

[0034] In one possible embodiment, the input image size is 84×84×3, that is, the width and height are 84 pixels and the number of channels is 3; the four labeled images (support set) and the one unlabeled image (query set) are passed through a first two-dimensional convolutional layer with a kernel size of 3, a padding value of 1, a stride of 1 and a channel size of 64 to obtain a feature map of size 83×83×64; then, by max pooling to reduce useless information, the feature map is reduced to a feature map F1 of size 41×41×64.

[0035] Understandably, two-dimensional convolutional layers are used for image feature extraction, dimensionality adjustment, and information exchange. Their specific calculation formula is as follows: N = (W - F + 2P) / S + 1, where N represents the output size, W is the input size, F is the kernel size, P is the padding size, and S is the stride size.

[0036] F1 = Max Pool(N), where Max Pool represents max pooling and F1 represents the output features of the first convolutional layer.

[0037] S2041B: Perform convolution calculation based on feature map F1 to obtain feature map F2.

[0038] The process is as follows Figure 11 As shown, the feature map F1 is input into the feature extraction module. In this invention, the feature extraction module can be a three-branch FENet module, which consists of a two-branch network and an attention mechanism in parallel. F1 is simultaneously input into both branches for feature learning and updating. A two-branch network and attention mechanism parallel module is used for feature extraction to obtain rich spatial details. This module consists of three different branches. A 1×1 convolution can be seen as a simple transformation for dimensionality reduction. After dimensionality reduction, 3×3 and 5×5 convolutions are performed sequentially to extract features into different receptive fields, obtaining a wider range of local spatial details. The other two branches are the channel attention module and the spatial attention module, which extract global feature information of the image from the channel and spatial aspects, respectively. The extraction flowchart is shown below. Figure 12 As shown, the channel attention module first performs global pooling and average pooling on the input feature map in the spatial dimension to compress the spatial size for easier learning of channel features. Then, the pooling results are fed into a multilayer perceptron (MLP) to learn the channel-dimensional features and the importance of each channel. Finally, the outputs are summed and processed by a sigmoid function to obtain the final result. The spatial attention module first performs global and average pooling on the input feature map in the channel dimension to compress the channel size for easier learning of subsequent spatial features. Then, the pooling results are concatenated according to the channels. Finally, the concatenated results are convolved and processed by an activation function to obtain the final result. In one possible embodiment, the specific module feature extraction process can be: feature map F1 is input into a convolution operation with a kernel size of 3, padding of 1, max pooling of 2, and 96 channels to extract feature map F2, which has a size of 20×20 and a dimension of 96.

[0039] S2041C. Based on feature map F2, different feature information is calculated and obtained through the feature extraction module, and the different feature information is summed to generate feature map F3.

[0040] Feature map F2 serves as the input to the dual-branch network and the multi-scale attention module. It is also input to three different branches of the module, which perform dual-branch convolution of size 3×3 and 5×5, spatial attention mechanism and channel attention mechanism respectively, to generate 4 feature information. The feature information is summed to obtain the extracted feature map F3.

[0041] In one possible implementation, such as Figure 5 The image classification method based on graph neural networks shown may include steps S2041C1-S2041C4.

[0042] S2041C1. Based on feature map F2, perform convolution calculations at the first and second scales respectively to obtain the feature information at the first scale F3. 1 Second-scale feature information F3 1 .

[0043] In one embodiment, the first scale is a 3×3 convolution kernel and the second scale is a 5×5 convolution kernel.

[0044] The feature map F2 is subjected to convolution operations with kernel sizes of 3×3 and 5×5 using a dual-branch network module to obtain feature information F3 at different scales. 1 The specific calculation formula is as follows: F3 1 =BN(Relu(Conv2d 3×3 (F2)+Conv2d 5×5 (F2))) Where BN is batch normalization, ReLU is the activation function, and Conv2d is a two-dimensional convolution operation.

[0045] S2041C2, The spatial attention mechanism module performs weighted calculations on each pixel in feature map F2 to obtain important region feature information F3. 2 .

[0046] The feature map F2 is input to the spatial attention mechanism module, which weights each pixel of the feature map, performs spatial transformation, and automatically captures features of important regions. 2 .

[0047] The specific calculation formula is as follows: F3 2 =σ(Mlp(AvgPool(F2))+Mlp(MaxPool(F2))) Where σ represents the Sigmoid function, MLP is a multilayer perceptron, F2 represents the input feature map, and AvgPool and MaxPool are the global pooling operation and the max pooling operation, respectively.

[0048] S2041C3, F3: Feature information is obtained by computation based on compressed network and excitation network. 3 .

[0049] The feature map F2 input channel attention mechanism module is divided into two networks: compression and activation. The purpose of the compression network is to compress the global information space, then learn the channel-dimensional feature information to form the importance of each channel, and finally assign different weights to each channel through the activation network.

[0050] The specific calculation formula is as follows: Where F2 represents the input feature map, F tr For convolution operations, u c To provide the feature information after convolution of the input feature map, z c For compression, after obtaining multiple feature maps u, a global average pooling operation is used to compress each feature map, transforming the C feature maps into a 1×1×C sequence; s is the activation operation, δ is the ReLU activation function, and σ is the sigmoid activation function; finally, a 1×1×C sequence is obtained and u c The product operation yields the final output F3. 3 .

[0051] S2041C4, Transfer the first-scale feature information F3 1 Second-scale feature information F3 1 Important regional feature information F3 2 and feature information F3 3 Summation is performed, and the summation result is used to calculate the feature map F3 based on the activation function and max pooling.

[0052] The one-scale feature information F3 obtained from steps S2041C1, S2041C2, and S2041C3 will be used. 1 Second-scale feature information F3 1 Important regional feature information F3 2 and feature information F3 3 Summing is performed, and the summation result is processed by activation function and max pooling to obtain feature map F3, which has a size of 10×10 and a dimension of 128. That is, F3 = Relu(MaxPool(F3) 1 +F3 2 +F33 )).

[0053] S2041D: Change the number of channels in the feature extraction module to the second and third channels respectively, and sum them to generate feature map F4.

[0054] In one embodiment, the number of second channels is 128 and the number of third channels is 256. Following the method in step S2041C, the number of channels is modified from 96 to 128 and 256 respectively for feature map extraction, and finally a feature map F4 with a size of 5×5 and a dimension of 256 is obtained.

[0055] S2041E calculates the features of sample images using a fully connected network structure.

[0056] The feature information F4 obtained in step S2041D is processed through a fully connected network structure to finally obtain 128 image features F of size 1×1.

[0057] S2042. Establish an initial connected graph, calculate the similarity values ​​between nodes in the initial connected graph based on the edge label graph neural network module, and obtain the updated connected graph.

[0058] An initial connected graph is established based on the image features extracted by the parallel module of the dual-branch network and attention mechanism. The connected graph is then fed into the edge label graph neural network module based on the node attention mechanism. Through the learning and updating of nodes and edge labels, the similarity between nodes can be obtained.

[0059] In one possible implementation, such as Figure 6 The image classification method based on graph neural networks shown may include steps S2042A-S2042B in step S2042.

[0060] S2042A. Establish an initial connected graph G based on the support set of the input sample image and the features of the queried sample image.

[0061] An initial fully connected graph G=(N,E;T) is constructed from the features F of the support set and query set samples of the input image; for example, Figure 10 As shown.

[0062] like Figure 10 As shown, each node represents a sample (solid line represents the support set S, dashed line represents the query set Q), and each edge represents the relationship connecting the nodes. G=(N,E;T) is a graph constructed from the feature set samples of nodes and edges of task T, where N=n_i, i=1,2,3⋯T, which is the node feature representation of the graph; E=[e ij ] 2 ∈[0,1] 2Let i,j=1,2,3⋯T. The features of the edges are represented in two dimensions, namely the features within the class and the features between the classes, with values ​​between 0 and 1, which are used to represent the probability of whether there is a relationship between two nodes.

[0063] The initial value n_i^0 of its nodes is extracted from the input samples by the feature extraction network, and the initial value e of the edges is... ij 0= , where || is the concatenation operator.

[0064] S2042B: Calculate the initial fully connected graph G based on the edge label graph neural network module, and obtain the updated connected graph.

[0065] In one possible implementation, such as Figure 7 The image classification method based on graph neural networks shown may include steps S2042B1-S2042B5.

[0066] S2042B1. Based on the node update network, calculate the node features and edge features in the initial fully connected graph G to obtain the updated node features; like Figure 2 As shown, the node update network includes an aggregation network, a connection network, and a multi-layer sensor. After entering the edge label graph neural network module, Figure G first updates the nodes by updating the input node features n. i 0 and edge features e ij 0 A node update network, composed of aggregation networks, connection networks, and multi-layer sensors, is used to update node features; updating node n i 1 The calculation formula is as follows: in , representing the weight of edge features among similar neighbors within the same node class, while This indicates the proportion of edge features between node classes (dissimilar neighbors); node updates are achieved by simultaneously performing aggregation and concatenation operations on intra-class and inter-class feature information within the node.

[0067] S2042B2. Calculate the updated edge label features based on the updated node feature information.

[0068] Based on the updated node features Information is used to update edge label features, through node information. To regain the similarity between nodes, i.e. and Then, based on the original side labels , And updated similarity information , This is used to update the feature information of each edge; its main module is a similarity network (SN) for intra-class similarity, which consists of four convolutional modules; the edge label update formula is shown below: in For a neural network of intra-class similarity, (1- This represents the dissimilarity between classes, with two-node information as input. , ,because Finally, a sigmoid function is used, which outputs either 0 or 1 to indicate whether a similarity relationship exists between nodes; this yields the current similarity results. , With the original side label , By combining these elements and calculating their correlation, the updated edge label information is obtained. .

[0069] S2042B3. Calculate the attention coefficient between node q in the query set and its neighbor node p.

[0070] The attention coefficient between query set node q and its neighbor node p is calculated using the following formula: in Perform a linear transformation on the nodes, || concatenate the transformed results of q and p, and map the concatenated features onto real numbers using a.

[0071] S2042B4. The attention coefficient is normalized and calculated according to the Softmax function.

[0072] Attention coefficient Perform Softmax normalization: , where Relu is the activation function, k∈S.

[0073] S2042B5. Calculate the attention coefficients of the query set nodes and their neighbor nodes after normalization based on the weighted summation, and obtain the updated query set node feature information.

[0074] The attention counts of the query node and its neighboring nodes are weighted and summed to obtain the updated query set node feature information: By repeating steps 2042B1, 2042B2, and 2042B3 twice, we can obtain the connected graph G after learning and updating.

[0075] S2043. Based on the updated connected graph, calculate the similarity value between the set node and other nodes using the Softmax function to obtain the classification result of the small sample image.

[0076] In one possible implementation, such as Figure 8 The image classification method based on graph neural networks shown may include steps S2043A-S2043B in step S2043.

[0077] S2043A: After updating the node and edge label feature information based on the three-layer edge label graph neural network module, the final node features and final edge label features are calculated and obtained.

[0078] After the three-layer edge label graph neural network module model updates the node and edge label feature information, the final node features can be obtained. and edge label features Therefore, query set nodes Classification can be performed using support set label and edge label feature results.

[0079] S2043B, Calculate the classification results of sample images based on the Softmax function.

[0080] The formula for calculating the Softmax function is: .

[0081] Figure 9 This is a schematic diagram of an image classification device based on a graph neural network, provided as an embodiment of this application. It is used to perform... Figure 2 The image classification method based on graph neural networks is shown. The graph neural network-based image classification device 300 includes: a configuration unit 301, a setting unit 302, and a calculation unit 303.

[0082] Configuration unit 301 is used to configure sample image classification tasks; each classification task includes a support set and a query set, the support set includes N classes of sample images, and each class of sample images includes label samples K; Configuration unit 301 is also used to configure a public dataset of sample images; the public dataset includes mini-ImageNet and tiered-ImageNet; mini-ImageNet includes 100 class images, each class includes 600 84×84 pixel RGB color images; tiered-ImageNet includes 34 superclass images, each superclass includes a first preset interval of subclass images; Setting unit 302 is used to set the learning parameters of the public dataset of sample images; the learning parameters include the initial learning rate, weight decay coefficient, mini-batch value of the meta-training task, and learning rate decay value; The computing unit 303 is used to calculate the classification results of the sample images based on the neural network model. The neural network model includes a feature extraction module and a side label graph neural network module. The feature extraction module is used to extract and fuse the local and global features of the input sample images by the neural network branches and attention branches. The side label graph neural network module is used to learn and update node feature information and side label feature information.

[0083] In the case of implementing the functions of the integrated modules described above in hardware, this application provides a possible structural schematic diagram of the electronic device involved in the above embodiments.

[0084] like Figure 1 As shown, an electronic device 100 is used to solve the problems of lack of key information in graph learning in subsequent graph modules of convolutional neural networks and poor criticality and completeness of generated image feature information due to the lack of good updating of node information during the graph learning process of graph neural networks based on edge labels, resulting in low image classification accuracy.

[0085] For example, used for execution Figure 2 This illustrates an image classification method based on graph neural networks. The electronic device 100 includes a processor 101, a memory 102, and a bus 103. The processor 101 and the memory 102 are connected via the bus 103. The processor 101 is the control center of the communication device and can be a single processor or a collective term for multiple processing elements. For example, the processor 101 can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor.

[0086] As one embodiment, processor 101 may include one or more CPUs, for example Figure 1CPU0 and CPU1 are shown in the diagram. Memory 102 may be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0087] As one possible implementation, the memory 102 can exist independently of the processor 101. The memory 102 can be connected to the processor 101 via the bus 103 and is used to store instructions or program code. When the processor 101 calls and executes the instructions or program code stored in the memory 102, it can implement the image classification method based on graph neural networks provided in this application embodiment.

[0088] In another possible implementation, memory 102 can also be integrated with processor 101. Bus 103 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 1 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0089] It should be pointed out that, Figure 1 The structure shown does not constitute a limitation on the electronic device 100. Except... Figure 1 In addition to the components shown, the electronic device 100 may include more or fewer components than those shown, or combine certain components, or have different component arrangements.

[0090] As an example, combined Figure 9 The functions implemented in the image classification device 300 based on graph neural networks are... Figure 1 The processor 101 in it has the same function.

[0091] Optional, such as Figure 1As shown, the electronic device 100 provided in this application embodiment may further include a communication interface 104. The communication interface 104 is used to connect with other devices via a communication network. This communication network may be Ethernet, a wireless access network, a wireless local area network (WLAN), etc. The communication interface 104 may include a receiving unit for receiving data and a transmitting unit for sending data.

[0092] In one possible implementation, the communication interface in the electronic device provided in this application embodiment can also be integrated into the processor.

[0093] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional units is used as an example. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0094] This application also provides a computer-readable storage medium storing instructions. When a computer executes these instructions, the computer performs each step of the method flow shown in the above-described method embodiments.

[0095] Embodiments of this application provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform an image classification method based on a graph neural network as described in the above-described method embodiments. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), registers, hard disks, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof, or any other form of computer-readable storage medium in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and storage medium may reside in an application-specific integrated circuit (ASIC). In the embodiments of this application, the computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device. Since the electronic devices, computer-readable storage media, and computer program products in the embodiments of this application can be applied to the above methods, the technical effects obtained can also be referred to the above method embodiments, and the embodiments of this application will not be repeated here.

[0096] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. An image classification method based on graph neural networks, characterized in that, The method includes: Configure sample image classification tasks; each classification task includes a support set and a query set, the support set includes N classes of sample images, and each class of sample images includes K label samples; Configure a public dataset of sample images; Set the learning parameters for the public dataset of sample images; the learning parameters include the initial learning rate, weight decay coefficient, mini-batch size of the meta-training task, and learning rate decay value; The classification result of the sample image is calculated based on a neural network model; the neural network model includes a feature extraction module and a side label graph neural network module. The feature extraction module is used to extract and fuse local and global features of the input sample image by neural network branches and attention branches. The side label graph neural network module is used for learning and updating node feature information and side label feature information. The classification result of the sample image calculated based on the neural network model includes: The features of the sample image are calculated and obtained based on the feature extraction module. An initial connected graph is established, and the similarity values ​​between nodes in the initial connected graph are calculated based on the edge label graph neural network module to obtain an updated connected graph. Based on the updated connected graph, the similarity value between the set node and other nodes is calculated using the Softmax function to obtain the classification result of the sample image; The feature extraction module includes three parallel modules and a fully connected layer. Each parallel module includes three two-dimensional convolutional layers, a spatial attention mechanism module, a channel attention mechanism module, five batch normalization layers, two activation functions, and a max pooling layer. The channel attention mechanism module includes a compression network and an activation network. The step of calculating and obtaining the features of the sample image based on the feature extraction module includes: The feature map F1 of the sample image is calculated and obtained based on the two-dimensional convolutional layer and the max pooling layer. Feature map F2 is obtained by performing convolution calculation based on feature map F1; Based on the feature map F2, different feature information is calculated and obtained by the feature extraction module, and the different feature information is summed to generate feature map F3; The number of channels in the feature extraction module is changed to the second number of channels and the third number of channels, respectively. Based on the feature map F2, different feature information is calculated and obtained by the feature extraction module. The different feature information is summed to generate feature map F4. The features of the sample images are calculated using a fully connected network structure; The process of establishing an initial connected graph, calculating the similarity values ​​between nodes in the initial connected graph based on the edge-labeled graph neural network module, and obtaining an updated connected graph includes: An initial fully connected graph G is constructed based on the support set of the input sample images and the features of the queried sample images; The initial fully connected graph G is calculated based on the edge label graph neural network module, and the updated connected graph is obtained. The step of calculating the initial fully connected graph G based on the edge-labeled graph neural network module and obtaining the updated connected graph includes: The node update network is used to calculate the node features and edge features in the initial fully connected graph G to obtain updated node features; the node update network includes an aggregation network, a connection network, and a multilayer sensor. Calculate the updated edge label features based on the updated node feature information; Calculate the attention coefficient between node q and its neighbor node p in the query set; The attention coefficients are normalized using the Softmax function; The updated query set node feature information is obtained by calculating the attention coefficients of the query set nodes and their neighbor nodes after normalization using a weighted summation.

2. The method according to claim 1, characterized in that, The step of calculating and obtaining different feature information based on the feature map F2 through the feature extraction module, and summing the different feature information to generate feature map F3 includes: Based on the feature map F2, convolution calculations are performed at the first and second scales respectively to obtain the feature information F3 at the first scale. 1 Second-scale feature information F3 1 ; Based on the spatial attention mechanism module, each pixel in the feature map F2 is weighted and calculated to obtain important region feature information F3. 2 ; Feature information F3 is calculated and obtained based on the compression network and the excitation network. 3 ; The first-scale feature information F3 1 Second-scale feature information F3 1 Important regional feature information F3 2 and feature information F3 3 Summation is performed, and the summation result is used to calculate the feature map F3 based on the activation function and the max pooling layer.

3. The method according to claim 1, characterized in that, The step of calculating the similarity value between the set node and other nodes using the Softmax function based on the updated connected graph to obtain the classification result of the sample image includes: After updating the node and edge label feature information based on the three-layer edge label graph neural network module, the final node features and final edge label features are calculated and obtained. The classification result of the sample image is calculated based on the Softmax function.

4. The method according to claim 1, characterized in that, The public datasets include mini-ImageNet and tiered-ImageNet; mini-ImageNet includes 100 categories of images, each category including 600 84×84 pixel RGB color images; tiered-ImageNet includes 34 superclass images, each superclass including a first preset interval of subclass images.

5. An image classification apparatus based on a graph neural network, the apparatus being used to perform the method as described in any one of claims 1-4, characterized in that, The device includes: a configuration unit, a setting unit, and a calculation unit; The configuration unit is used to configure sample image classification tasks; each classification task includes a support set and a query set, the support set includes N classes of sample images, and each class of sample images includes label samples K; The configuration unit is also used to configure a public dataset of sample images; The setting unit is used to set the learning parameters of the public dataset of sample images; the learning parameters include the initial learning rate, weight decay coefficient, mini-batch value of the meta-training task, and learning rate decay value; The computing unit is used to calculate the classification result of the sample image based on a neural network model. The neural network model includes a feature extraction module and a side label graph neural network module. The feature extraction module is used to extract and fuse local and global features of the input sample image by neural network branches and attention branches. The side label graph neural network module is used to learn and update node feature information and side label feature information.

6. An electronic device, characterized in that, The electronic device includes: A processor and a memory; the memory stores instructions executable by the processor; When the processor is configured to execute the instructions, it causes the electronic device to implement the method as claimed in any one of claims 1-4 or the apparatus as claimed in claim 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes: computer software instructions; When the computer software instructions are executed in an electronic device, the electronic device causes the electronic device to implement the method as claimed in any one of claims 1-4 or the apparatus as claimed in claim 5.

Images