A small sample semantic segmentation method based on network motif graph representation learning

Abstract

The present application belongs to the field of artificial intelligence research, and proposes a small sample semantic segmentation method based on network motif graph representation learning. The present application mainly solves the problem of channel information loss in the query image caused by insufficient channel feature extraction capability. The method mainly includes a feature extraction module, a relationship reference module, an implicit relationship mining module, a multi-scale interaction module and a final segmentation module. The present application takes the query channel feature as a node, constructs a graph structure, and establishes the relationship between the nodes; the network motif is used to quantify the attribute features and structural features of the nodes to enhance the relationship between the channels; finally, the attribute features and structural features are aggregated, and the implicit relationship between the nodes is mined through graph representation learning. The present application displays the implicit relationship of the query channel feature, which alleviates the information loss problem in the query image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence research, and in particular to a small sample semantic segmentation method based on network motif graph representation learning. BACKGROUND

[0002] Image semantic segmentation belongs to the computer vision recognition task, and plays an important role in the field of artificial intelligence. At present, it is mainly applied in the fields of medical images, remote sensing images and unmanned driving. The image semantic segmentation task is essentially to classify and locate each pixel point in the image. This task is also considered to be one of the dense prediction tasks. However, this task needs a large amount of training data set with dense annotation to support, and the collection cost of these data sets is high and time-consuming, and the current method is difficult to effectively promote to the unobservable object class in a data effective way. In order to solve the data problem, small sample semantic segmentation is proposed. The goal of small sample semantic segmentation in “One-shot learning for semantic segmentation,” in British Machine Vision Conference (BMVC) is to segment the foreground image of an unobservable class in the query image using one or several annotated support images, wherein the unobservable class is contained in both the support image and the query image.

[0003] The small sample semantic segmentation method is generally expressed in “Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning,” in IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). A double-branch convolutional neural network is used [2]. The support branch extracts the segmentation prior of the support image, and the query branch extracts the feature of the query image in the embedding space. According to the segmentation prior of the support branch, the similarity relationship between the support image and the query image is constructed to realize the segmentation of the query image. On this basis, the existing methods fully mine the support image information and the query image information by generating a support prototype in the support branch, calculating the similarity between the support feature and the query feature by different methods, or fusing the multi-scale features of the two branches to improve the segmentation performance. However, these methods only mine and utilize the spatial feature information in the initial feature extraction and subsequent interaction process, and their extraction ability of channel feature information is insufficient. SUMMARY

[0004] The existing small sample semantic segmentation method lacks a method for modeling channel feature display, and the ability to extract channel feature information is insufficient, resulting in the problem of loss of channel information in the query image. The present application proposes a small sample semantic segmentation method based on network motif graph representation learning.

[0005] The technical scheme of the present application: a small sample semantic segmentation method based on network motif graph representation learning, taking the channel features of the query image as nodes, constructing a graph structure, and establishing the relationship between the nodes. The network motif is used to quantify the attribute features of the nodes and the structural features of the nodes to enhance the relationship between the channels. The attribute features and the structural features are aggregated through graph representation learning to mine the implicit relationship between the nodes. This method displays the implicit relationship of the query channel features, thereby alleviating the information loss problem in the query image. The small sample semantic segmentation method based on network motif graph representation learning includes a feature extraction module, a relationship reference module, an implicit relationship mining module, a multi-scale interaction module, and a final segmentation module, and the specific steps are as follows:

[0006] S1: The feature extraction module extracts the features of the support image and the query image, and generates a support prototype:

[0007] The feature extraction backbone network is used to extract the features of the support image and the query image, which are support features and query features respectively. The support features and the query features are divided into intermediate features and high-level features. The mask global pool is used on the intermediate support features to remove background interference and obtain the support prototype. The intermediate features reveal the invisible class-shared object parts, such as color, position, and edge. The high-level features are related to object categories, but will have an adverse effect on the final segmentation result. However, the semantic information provided by the high-level features is very important. The high-level features are used to establish the relationship between the support features and the query features, providing prior knowledge for subsequent segmentation, which solves the above problems.

[0008] S2: The relationship reference module calculates the correlation between the support features and the query features to provide prior knowledge for subsequent segmentation:

[0009] The relationship between the high-level support features obtained in step S1 and the query features is calculated to generate a relationship reference graph.

[0010] S3: The implicit relationship mining module models the channel features of the query image as a topological graph, and mines the implicit channel relationship of the query image based on network motif graph representation learning:

[0011] The implicit relationship of the intermediate query features obtained in step S1 is mined. The discrete channel features in the query image are converted into a topological graph. Through network motif graph representation learning, the beneficial implicit correlation of the channel features of the intermediate query features is extracted to obtain new query features.

[0012] Information features include spatial features and channel features, which play different but indispensable roles in small sample semantic segmentation. Current methods usually only use spatial features, lacking the ability to extract channel features. The invention proposes an implicit relationship mining module to model discrete channel features into channel feature maps, and the relationship between channel features can be learned by using the topology of the graph. This step can extract the implicit correlation of channel features and enrich the information of the query image.

[0013] S4: The multi-scale interaction module fuses the outputs of the feature extraction module, the relationship reference module, and the implicit relationship mining module based on multi-scale:

[0014] The relationship reference graph, the new query feature, and the support prototype are connected; based on the multi-scale interaction module, the new query feature is adaptively enriched by the relationship reference graph and the support prototype, and the fused query feature is obtained;

[0015] The size, position, and pose of the query image will be very different from the support image, resulting in spatial inconsistency. The invention uses a multi-scale interaction module, in the horizontal direction, the new query feature, the support prototype, and the relationship reference graph interact and merge; in the vertical direction, the multi-scale interaction module integrates more refined features into the new query feature output by the implicit relationship mining module from top to bottom; finally, the features of different scales are collected as the final fused query feature; this step can solve the problem of spatial inconsistency.

[0016] S5: The final segmentation module predicts the query image mask and segments by fusing the query feature: using a classification header on the fused query feature for segmentation to obtain the final query image prediction mask;

[0017] S6: Calculate the loss between the query image real mask and the final query image prediction mask using the cross-entropy loss function, and the loss between the query image real mask and the query image prediction mask under different scales, add the two losses, and update the model parameters using the gradient backpropagation function.

[0018] The step S1 is specifically:

[0019] S11: Use a convolutional neural network as the backbone network to extract features from the support image and the query image to obtain the intermediate support feature X sm , the intermediate query feature X qm , the high-level support feature X sh , and the high-level query feature X qh ; the dimensions of the obtained support features and query features are both HxWxC, where H and W represent the height and width of the features, respectively, and C represents the number of channels;

[0020] S12: The intermediate support feature X smand support prototype X′ is obtained by multiplying support image mask s ;

[0021] Wherein, the query image is a picture, the support image is one or more pictures, and all images have corresponding mask images.

[0022] The step S2 is specifically:

[0023] S21: Calculate the cosine similarity between the high-level query features and the high-level support features X sh obtained in step S1.

[0024] S22: Calculate the cosine similarity between each high-level query feature and the high-level support features X sh obtained in step S1.

[0025] S23: Normalize all relationship values to obtain relationship reference map R q ; the dimension of the relationship reference map is HxWx1.

[0026] The step S3 is specifically:

[0027] S31: Map the pixel-level middle-level query features X qm to channel features X g by the following method:

[0028] X g = f 1×1 (X qm )·f 1×1 (X qm ) T

[0029] Wherein, the channel features X g have a node feature dimension of NxD in the topological graph, N represents the number of nodes in the topological graph, D represents the dimension of each node feature, and f 1×1 () represents a 1x1 convolution.

[0030] S32: Use k-neighbor graph to construct the adjacent relationship between nodes and obtain adjacency matrix A g .

[0031] First, calculate the similarity between N nodes to obtain a similarity matrix, and then select the k nodes with the highest similarity for each node as its neighbor nodes; the nodes and their k-neighbor nodes are connected by an undirected edge to obtain a symmetric adjacency matrix A g ; the topological graph G a =(A g ,X g ) is obtained.

[0032] S33: Calculate the node motif degree of each network motif for each node as the attribute information of the node;

[0033] The network motif is a connected network subgraph; the top five network subgraphs with the highest occurrence frequency are selected as the network motif; the node motif degree is used to define the number of each network motif contained in each node in the topological graph; the motif degree of five network motifs of each node is calculated and used as the attribute information of the node, which is the attribute feature tensor X aft ;

[0034] S34: Select three network motifs with the highest node motif degree as the structure information, and represent it as the structure feature tensor X sft ;

[0035] S35: Input the attribute feature tensor and the structure feature tensor obtained in steps S33 and S34 into two graph convolution networks respectively to obtain the attribute embedding tensor and the structure embedding tensor;

[0036] The single-layer graph convolution network is defined as:

[0037] G=σ(AX g W)

[0038] Wherein, σ(·) is a nonlinear activation function, and W is a weight matrix; in the present application, the RELU function is used as the activation function;

[0039] S36: Aggregate the attribute embedding tensor and the structure embedding tensor obtained in step S35 by using Hadamard product to obtain the aggregated query image channel feature X crm ; The Hadamard product aggregation matrix requires that the elements are multiplied by each other, and it has the same dimension as the attribute embedding tensor and the structure embedding tensor;

[0040] S37: Map the aggregated query image channel feature back to the original coordinate space by the following formula:

[0041]

[0042] At this time, the dimension of the new query feature is HxWxC.

[0043] The step S4 is specifically:

[0044] S41: Connect the relationship reference graph, the new query feature and the support prototype to generate a merged feature X m :

[0045]

[0046] S42: Adjust X m to n different sizes by using an interpolation function, and obtain

[0047] S43: Interact between different scales to achieve feature fusion:

[0048]

[0049] Features at the current scale The characteristics of the previous scale are Connect them, and the two connected layers use a 1×1 convolutional layer; then use two 3×3 convolutional layers F. α To achieve feature fusion at each scale; using 1×1 convolutions and 3×3 convolutions Fi α Use skip connections when necessary;

[0050] S44: Use a classifier to perform mask prediction on the refined features at each scale to obtain the predicted mask of the query image at different scales;

[0051] S45: After adjusting the refined features at multiple scales to the same scale through an interpolation function, they are connected together and then processed by a 1×1 convolution to generate fused query features.

[0052] Specifically, step S5 involves segmenting the fused query features using the same classifier as in step S44 to obtain the final query image prediction mask.

[0053] The beneficial effects of this invention are as follows: The few-sample semantic segmentation method based on network motif graph representation learning provided by this invention models the channel features of the query image as a channel topology graph, quantifies node features through network motifs to enhance node relevance, and makes implicit channel feature relationships explicit through graph representation learning, thereby improving channel feature extraction capabilities and alleviating the problem of query image information loss. Extensive experiments were conducted on the PASCAL-5i and FSS-1000 datasets, and the experimental results are superior to existing methods. Attached Figure Description

[0054] Figure 1 This is a basic framework diagram of the few-sample semantic segmentation method based on network motif graph representation learning of this invention. Detailed Implementation

[0055] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0056] A small sample semantic segmentation method based on network motif graph representation learning mainly includes five parts, a feature extraction module extracts the features of the support image and the query image and generates a support prototype; a relationship reference module calculates the correlation between the support features and the query features to provide prior knowledge for subsequent segmentation; an implicit relationship mining module models the query channel features as a topological graph, and mines the implicit channel relationship of the query image through network motif-based graph representation learning; a multi-scale interaction module fuses the outputs of all the above modules based on multi-scale; and a final segmentation module predicts the query image mask and segments the fused query features. For example Figure 1 The specific steps are as follows:

[0057] S1: Feature extraction module: use a feature extraction backbone network to extract middle-level features and high-level features of the support image and the query image; use a mask global pool on the middle-level support features to remove background interference and obtain a support prototype.

[0058] Wherein the query image is one picture, the support image is one or more pictures, and all images have corresponding mask images.

[0059] S11: Use a convolutional neural network as a backbone network to extract features of the support image and the query image to obtain middle-level support features X sm , middle-level query features X qm , high-level support features X sh and high-level query features X qh .

[0060] For example, ResNet-50 and VGG-16 are used as backbone networks, and the support branch and the query branch share the backbone network parameters. The backbone network is generally divided into four groups: block1, block2, block3 and block4, each of which outputs feature information. In this embodiment, the features output by block2 and block3 are connected, and the channels are compressed to a fixed dimension of 256 through 1x1 convolution as middle-level features; the features output by block4 are used as high-level features, and the channel number is also 256.

[0061] In the present application, the dimensions of the support features and the query features obtained in step S1 are both HxWxC, wherein H and W represent the height and width of the features, and C represents the number of channels.

[0062] S12: Multiply the middle-level support features and the support image mask to obtain the support prototype X' s .

[0063] S2: Relationship reference module: calculate the relationship between the high-level support features and the query features obtained in step S1 to generate a relationship reference graph.

[0064] S21: Calculate the cosine similarity between the advanced support features and the advanced query features obtained in step S1.

[0065] S22: For each advanced query feature vector, calculate the highest value of its cosine similarity with the advanced support features as the relationship value.

[0066] S23: Normalize all relationship values to obtain the relationship reference graph R q .

[0067] where the dimension of the relationship reference graph is HxWx1.

[0068] The middle-level features reveal the object parts shared by invisible classes, such as color, position, edge, etc. The high-level features are related to the object class, but will have an adverse effect on the final segmentation result. However, the semantic information provided by the high-level features is very important. This step uses high-level features to establish the relationship between support features and query features, providing prior knowledge for subsequent segmentation and solving the above problems.

[0069] S3: Implicit relationship mining module: Mine the implicit relationship of the middle-level query features obtained in step S1. Convert the discrete channel features into a topological graph; extract the beneficial implicit correlation of the channel features of the query features through network motif-based graph representation learning, and obtain new query features.

[0070] First, a topological graph G a = (A g , X g ) needs to be constructed. Where A g is the adjacency matrix, and X g is the node feature.

[0071] S31: Map the pixel-level middle-level query features X qm to channel-level features X g in the following way:

[0072] X g = f 1×1 (X qm )·f 1×1 (X qm ) T

[0073] where the channel-level feature X g has a dimension of NxD, N represents the number of nodes in the graph, and D represents the dimension of each node feature; f 1×1 () represents a 1x1 convolution.

[0074] S32: Use k-neighbor graph to construct the adjacency relationship between nodes and obtain the adjacency matrix A g .

[0075] First, the similarity between N nodes is calculated to obtain a similarity matrix, and then the k nodes with the highest similarity are selected as the neighbor nodes of each node.

[0076] But the adjacency relationship obtained by this method is asymmetric. In order to obtain a symmetric adjacency matrix, the direction of the edge is ignored, that is, if node y is the k-neighbor of node x, then x and y are connected by an undirected edge. Finally we obtain a symmetric adjacency matrix A g .

[0077] Next, the feature information of the graph is calculated, mainly including the attribute information and the structure information of the nodes.

[0078] S33: Calculate the node motif degree of each network motif for each node, and take it as the attribute information of the node.

[0079] Network motif refers to a special connected network subgraph. This step selects five high-frequency network subgraphs as network motifs. The node motif degree is used to define the number of each network motif contained by each node in the graph. This step calculates the motif degree of five network motifs for each node, and takes it as the attribute information of the node, which is represented as the attribute feature tensor X aft .

[0080] S34: Select three motifs with the highest node motif degree as structure information, which is represented as the structure feature tensor X sft .

[0081] By quantifying the attribute information and the structure information of the nodes through network motifs, the beneficial channel features can be enhanced and the correlation between the implicit channel features can be represented.

[0082] Finally, the attribute features and the structure features are aggregated. There is a big difference between the attribute feature tensor and the structure feature tensor obtained in steps S33 and S34, which can easily cause the problem of uneven dimension in subsequent calculation. At the same time, in order to express the similarity between the abstract features of similar objects in the image, the graph convolution network is adopted.

[0083] S35: The attribute feature tensor and the structure feature tensor obtained in steps S33 and S34 are input into two graph convolution networks respectively to obtain an attribute embedding tensor and a structure embedding tensor.

[0084] Generally, a single-layer graph convolution network is defined as:

[0085] G=σ(AXW)

[0086] Where σ(·) is a nonlinear activation function, and W is a weight matrix; the RELU function is used as the activation function.

[0087] S36: Obtain the aggregated query image channel feature X by using the Hadamard product of the two embedded tensors obtained in step S35 crm The Hadamard product aggregation matrix requires elements to multiply each other, so it has the same dimension as the original matrix.

[0088] S37: Map the aggregated channel feature back to the original coordinate space through the following formula:

[0089]

[0090] At this time, the new query feature has a dimension of HxWxC.

[0091] Information features include spatial features and channel features, which play different but indispensable roles in small sample semantic segmentation. The current method lacks the ability to extract channel features by only using spatial features. The present application proposes an implicit relationship mining module to model discrete channel features into channel feature maps, and the relationship between channel features can be learned by using the topological structure of the graph. This step can extract the implicit correlation of channel features and enrich the query image information.

[0092] S4: Multi-scale interaction module: connect the relationship reference graph, the new query feature and the support prototype, and adaptively enrich the query feature based on multi-scale through the relationship reference graph, the new query feature and the support prototype to obtain the fused query feature.

[0093] S41: Connect the relationship reference graph, the new query feature and the support prototype to generate the merged feature X m :

[0094]

[0095] S42: Adjust X m to n different sizes using an interpolation function, and obtain

[0096] S43: Interact between different scales to realize feature fusion:

[0097]

[0098] Connect the current scale feature and the last scale feature , and then use a 1x1 convolution layer on them. The purpose of this is to transfer the coarse features of the previous scale to the current scale. Then, use two 3x3 convolutions F α to complete the feature fusion of each scale. At the same time, in order to better fuse all the characteristics, use a skip connection at 1x1 convolution and F α .

[0099] S44: Mask prediction is performed on the refined features of each scale using the classifier to obtain the predicted mask of the query image at different scales.

[0100] S45: The refined features of multiple scales are adjusted to the same scale through an interpolation function, and they are connected to generate the fused query feature through a 1x1 convolution.

[0101] The size, position and pose of the query image will be very different from the support image, resulting in spatial inconsistency. The present application uses a multi-scale interaction module. In the horizontal direction, the new query feature, the support prototype and the relationship reference map interact and merge. In the vertical direction, the multi-scale interaction module integrates more refined features into the new query feature output by the implicit relationship mining module from top to bottom. Finally, the multi-scale interaction module collects features of different scales as the final feature. This step can solve the problem of spatial inconsistency.

[0102] S5: The fused query feature is segmented using the classifier to obtain the final predicted mask of the query image.

[0103] S6: The cross-entropy loss function is used to calculate the loss between the real mask of the query image and the final predicted mask, and the loss between the real mask of the query image and the predicted mask at different scales, and the sum of them is used to update the model parameters using the gradient backpropagation function.

[0104] The present application has carried out experiments on two data sets of PASCA-5i and FSS-1000. PASCA-5i includes annotations of PASCAL VOC and SDS, and is a specially designated data set for small sample semantic segmentation. It contains 20 classes, which are divided into 4 groups on average, each group containing 5 classes. FSS-1000 is a data set specially used for small sample semantic segmentation, which contains 520 training classes, 240 validation classes and 240 test classes.

[0105] The present application uses the average intersection over union (mIoU) evaluation method to evaluate the performance, and the higher the mIoU value, the better the performance. The present application uses ResNet-50 and VGG-16 as the backbone network to carry out experiments on the PASCA-5i data set, and uses ResNet-50 as the backbone network to carry out experiments on the FSS-1000 data set. At the same time, experiments are carried out for one support image (1-shot) and five support images (5-shot). The experimental results are shown in Table 1 and Table 2.

[0106] Table 1 Experimental results of PASCA-5i data set

[0107]

[0108] As can be seen from Table 1, for 1-shot segmentation, when ResNet-50 is used as the backbone network, the mIoU reaches 61.0%, which is increased by 1.3% compared with the baseline. When VGG-16 is used as the backbone network, the mIoU reaches 55.9%. For 5-shot segmentation, when ResNet-50 is used as the backbone, the mIoU reaches 62.9%, which is increased by 1.4% compared with the baseline. When VGG-16 is used as the backbone, the mIoU reaches 60.4%. And the experimental results of the present application are all better than the existing CANet, PGNet, PFENet, PMMs, BriNet and other small sample semantic segmentation methods.

[0109] Table 2 FSS-1000 dataset experimental results

[0110]

[0111] As can be seen from Table 2, for 1-shot segmentation, when ResNet-50 is used as the backbone network, the mIoU reaches 74.8%, which is increased by 1.1% compared with the baseline. For 5-shot segmentation, when ResNet-50 is used as the backbone network, the mIoU reaches 75.7%, which is increased by 1.2%.

[0112] These results show that the method of using a graph with structural characteristics to learn channel implicit features proposed by the present application is effective for improving segmentation performance. At the same time, the present application also conducts sufficient ablation experiments to verify the necessity of each part in the implicit relationship mining module, and the correctness of selecting k = 3 when constructing a graph structure using a k-neighbor graph.

Claims

1. A small sample semantic segmentation method based on network motif graph representation learning, characterized in that, The small sample semantic segmentation method based on network motif graph representation learning constructs a graph structure by taking the channel features of the query image as nodes, quantizes the attribute features and structural features of the nodes by using the network motif, aggregates the attribute features and structural features by graph representation learning, and mines the implicit relationship between the nodes, thereby relieving the information loss problem in the query image. The small sample semantic segmentation method based on network motif graph representation learning includes a feature extraction module, a relationship reference module, an implicit relationship mining module, a multi-scale interaction module, and a final segmentation module, and the specific steps are as follows: S1. The feature extraction module extracts the features of the support image and the query image, and generates a support prototype: The features of the support image and the query image are extracted by using a feature extraction backbone network, and the features are support features and query features respectively. The support features and the query features are divided into intermediate features and high-level features, the intermediate support features represent the object parts shared by invisible classes, including color, position and edge, and the high-level features are related to object categories. The support prototype is obtained by using a support image mask global pool to remove background interference. S2. The relationship reference module is used to calculate the correlation between the support features and the query features, and to provide prior knowledge for subsequent segmentation: The relationship between the high-level support features obtained in step S1 and the query features is calculated to generate a relationship reference graph. S3. The implicit relationship mining module models the channel features of the query image as a topological graph, and mines the implicit channel relationship of the query image by network motif graph representation learning: The implicit relationship of the intermediate query features obtained in step S1 is mined. The discrete channel features in the query image are converted into a topological graph. The beneficial implicit correlation of the channel features of the intermediate query features is extracted by network motif graph representation learning to obtain new query features. S4. The multi-scale interaction module fuses the outputs of the feature extraction module, the relationship reference module, and the implicit relationship mining module based on multi-scale:

2. The network motif graph representation learning based few-shot semantic segmentation method according to claim 1, characterized in that, The relationship reference graph, the new query features, and the support prototype are connected. S11: a convolutional neural network is used as a backbone network to perform feature extraction on the support image and the query image to obtain middle-level support features middle-level query features high-level support features and high-level query features The dimensions of the obtained support features and query features are both HxWxC, where H and W represent the height and width of the features respectively, and C represents the number of channels. S12: multiply the support image mask by the support prototype and support image mask ; The new query features are adaptively enriched by the relationship reference graph and the support prototype based on the multi-scale interaction module to obtain fused query features.

3. The network motif graph representation learning based few-shot semantic segmentation method according to claim 1 or 2, characterized in that, In the horizontal direction, the new query features, the support prototype, and the relationship reference graph interact and merge. S21: Calculate the cosine similarity between the advanced support features obtained in step S1 and the advanced query features and the advanced query features; S22: compute for each high-level query feature its relation value with the high-level support feature the highest value of the cosine similarity as the relation value; S23: normalize all the relationship values to obtain a relationship reference map The dimension of the relationship reference map is HxWx1.

4. The network motif graph representation learning based few-shot semantic segmentation method according to claim 3, characterized in that, In the vertical direction, the multi-scale interaction module integrates more refined features into the new query features output by the implicit relationship mining module from top to bottom. Finally, the features of different scales are collected as the final fused query features. S5. The final segmentation module predicts the query image mask and segments by using the fused query features: The classification header is used to segment the fused query features to obtain the final query image prediction mask. S6. The cross-entropy loss function is used to calculate the loss between the query image real mask and the final query image prediction mask, and the loss between the query image real mask and the query image prediction mask under different scales. The two losses are added together, and the gradient backpropagation function is used to update the parameters of the model. The step S1 is specifically: Wherein, the query image is an image, the support image is one or more images, and all images have corresponding mask images. The step S2 is specifically: The step S3 is specifically: S31: Map the pixel-level mid-level query features to channel features by : ； wherein the channel feature The node feature dimension of the topological graph is NxD, N represents the number of nodes in the topological graph, and D represents the dimension of each node feature, represents a 1x1 convolution; S32: Construct the adjacent relationship between nodes by using the k-neighbor graph, and obtain the adjacency matrix ; First, the similarity between N nodes is calculated to obtain a similarity matrix, and then the k nodes with the highest similarity are selected as the neighbor nodes of each node; the nodes and their k-neighbor nodes are connected by an undirected edge to obtain a symmetric adjacency matrix ; obtain a topology graph =( ) S33: calculate the node motif degree of each network motif for each node as the attribute information of the node; The network motif is a connected network subgraph; the first five network motifs with the highest occurrence frequency are selected as the network motif; the node motif degree is used to define the number of each network motif contained in each node in the topological graph; the motif degree of the five network motifs of each node is calculated and used as the attribute information of the node, which is the attribute feature tensor ​ S34: select three network motifs with the highest node motif degree as structure information, denoted as structure feature tensor ; S35: input the attribute feature tensor and the structure feature tensor obtained in steps S33 and S34 into two graph convolution networks respectively to obtain an attribute embedding tensor and a structure embedding tensor; The single-layer graph convolution network is defined as: ； wherein, is a non-linear activation function, W is a weight matrix; the RELU function as an activation function; S36: Obtain the aggregated query image channel feature by using the Hadamard product of the attribute embedding tensor and the structure embedding tensor obtained in step S35 ; the Hadamard product aggregation matrix requires that the elements are multiplied with each other, and has the same dimension as the attribute embedding tensor and the structure embedding tensor; S37: map the aggregated query image channel features back to the original coordinate space through the following formula: ； New query features at this time with dimensions H x W x C.

5. The network motif graph representation learning based few-shot semantic segmentation method according to claim 4, characterized in that, The step S4 is specifically: S41: connecting the relationship reference graph, the new query feature and the support prototype to generate a merged feature : ； S42: using the interpolation function to obtain adjusted to n different sizes, and obtained ; S43: interact between different scales to realize feature fusion: ； features of the current scale and features of the previous scale are concatenated, and the concatenated features are processed using a 1x1 convolutional layer; Two 3x3 convolutions are used to complete the feature fusion of each scale. ; Using 1x1 and 3x3 convolutions When using skip connections; S44: use a classifier to perform mask prediction on the refined features of each scale to obtain a query image prediction mask at different scales; S45: after adjusting the refined features of multiple scales to the same scale through an interpolation function, connect them together to generate a fused query feature through a 1x1 convolution.

6. The network-motif graph representation learning based few-shot semantic segmentation method according to claim 4 or 5, characterized in that, The step S5 is specifically: use the same classifier as in step S44 on the fused query feature to obtain a final query image prediction mask.

Images