Image semantic segmentation method

Abstract

The invention relates to the field of computer vision, in particular to an image semantic segmentation method. According to the method, a feature graph corresponding to each image is extracted through an encoder, each image serves as a node, a connection relation is established between the nodes, and a completely directed graph is constructed; inputting the completely directed graph and all the feature graphs into a graph neural network, and extracting semantic similarity feature vectors; inputting the feature map corresponding to each image into an intra-class reasoning module, extracting class features of each class in each image, inputting the class features into a co-occurrence relation reasoning module, and extracting a co-occurrence feature vector of each image; and respectively splicing the semantic similarity feature vector and the co-occurrence feature vector of each image, calculating an adaptive weight through a full connection layer and a Softmax activation function, carrying out weighted calculation on the spliced feature vector according to the adaptive weight, and outputting a semantic segmentation feature vector of each image through the full connection layer. According to the method, the context information in the image is fully utilized, and the semantic segmentation precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image semantic segmentation method. Background Art

[0002] Image semantic segmentation is a key computer vision task, which aims to classify each pixel in an image into a specific semantic category, including objects (bicycles, cars, pedestrians) and the scenes they are in (road, sky, benches). By accurately classifying each pixel, semantic segmentation can provide fine-grained information for understanding and processing image content, which is of great significance in many application scenarios, such as autonomous driving, robotic vision, and medical image analysis.

[0003] Understanding contextual information is crucial when performing semantic segmentation. Contextual information refers not only to the types of objects in the image, but also to the relationships and interactions between these objects and their surroundings. For example, bicycles usually appear on roads, while pedestrians may appear on sidewalks or in parks. This contextual relationship provides key information to help the model more accurately identify and classify different objects and scenes. In complex scenes, factors such as the layout, size, and relative position of objects will affect the accuracy of segmentation.

[0004] However, existing semantic segmentation methods often only consider image-level contextual information or semantic-level contextual information. Such methods that only consider image-level contextual information mainly rely on large-scale information in the image for semantic segmentation. However, such methods are often not sufficient to cope with complex scenes with rich details in practical applications. Although image-level contextual information can provide an overall perspective, it is difficult to effectively describe the relationship and subtle differences between local objects, which affects the segmentation accuracy. Such methods that only consider semantic-level contextual information focus on the relationship between objects and their classifications, and only rely on semantic-level contextual information, often ignoring the complexity of the overall image environment. Such processing may result in the model being unable to fully understand and segment different elements in the image when objects interact with each other. Therefore, only considering image-level contextual information or semantic-level contextual information, this single-dimensional contextual information processing method limits the performance of the model in complex scenes, resulting in poor image semantic segmentation accuracy. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the defect that the existing semantic segmentation method only considers the context information at the image level or the context information at the semantic level, resulting in poor semantic segmentation accuracy.

[0006] In order to solve the above technical problems, the present invention provides an image semantic segmentation method, comprising the following steps:

[0007] Constructing a dual graph structure semantic aggregation network, wherein the dual graph structure semantic aggregation network includes: an encoder, a graph neural network, an intra-class reasoning module, and a co-occurrence relationship reasoning module;

[0008] Input multiple images including at least one common label into the encoder, and extract the feature map corresponding to each image;

[0009] Each image is taken as a node, and a connection relationship is established between each node to form a directed edge between each node. Based on each node and the edge between each node, a complete directed graph is constructed, in which each node includes a self-loop edge;

[0010] The fully directed graph and the feature map corresponding to each image are input into the graph neural network to extract the semantic similarity feature vector;

[0011] The feature map corresponding to each image is input into the intra-class reasoning module to extract the category features of each category in each image, including:

[0012] Pass the feature map corresponding to the current image through the convolution layer to obtain the decoupled representation of each category in the current image;

[0013] The decoupled representation of each category in the current image is downsampled and upsampled in turn to obtain an adaptive feature representation of each category in the current image;

[0014] The adaptive feature representation of each category in the current image is used through the self-attention mechanism to capture long-distance dependencies and obtain the category features of each category in the current image;

[0015] The category features of each category in each image are input into the co-occurrence relationship reasoning module to extract the co-occurrence feature vector of each image;

[0016] After concatenating the semantic similarity feature vector with the co-occurrence feature vector of each image, a concatenated feature vector of each image is obtained;

[0017] The concatenated feature vector of each image is passed through the fully connected layer and the Softmax activation function in turn to calculate the adaptive weight of each image;

[0018] The concatenated feature vector of each image is weighted according to the adaptive weight of each image, and the weighted feature vector of each image is passed through the fully connected layer to output the semantic segmentation feature vector of each image;

[0019] The semantic segmentation feature vector of each image is passed through the classifier, Softmax activation function, and argmax operation in turn to obtain the semantic segmentation map of each image.

[0020] Preferably, the step of inputting the category features of each category in each image into the co-occurrence relationship reasoning module to extract the co-occurrence feature vector of each image comprises:

[0021] Get the word vector of each category in the training set, and embed the word vector of each category in the training set into the feature space as a semantic concept node;

[0022] Based on the total number of times each category in the training set appears in the training set and the number of times each category co-occurs with other categories in the training set, a co-occurrence probability transfer matrix is ​​constructed;

[0023] For each image, the category features of each category in the current image are used as the category feature nodes of the current image;

[0024] Based on the co-occurrence probability transfer matrix, each semantic concept node and each category feature node of the current image, a weighted similarity matrix of the current image is constructed;

[0025] Based on the weighted similarity matrix of the current image, the semantic-category guidance matrix of the current image is calculated;

[0026] According to the semantic-category guidance matrix of the current image and each semantic concept node, each enhanced category feature node of the current image is calculated;

[0027] Based on each semantic concept node and each category feature node of the current image, the category-semantic guidance matrix of the current image is calculated;

[0028] According to the category-semantic guidance matrix of the current image and each category feature node of the current image, each enhanced semantic concept node is calculated;

[0029] Establish a bidirectional connection relationship between each enhanced semantic concept node and each enhanced category feature node of the current image to form a directed edge between each node, and construct a dual interactive directed graph of the current image based on each enhanced semantic concept node, each enhanced category feature node of the current image and the edges between each node;

[0030] The dual interaction directed graph, semantic-category guidance matrix and category-semantic guidance matrix of the current image are input into the graph neural network to obtain the co-occurrence feature vector of the current image.

[0031] Preferably, the constructing of a co-occurrence probability transfer matrix based on the total number of times each category in the training set appears in the training set and the number of times each category co-occurs with other categories in the training set comprises:

[0032] Based on the total number of times each category appears in the training set and the number of times each category co-occurs with other categories in the training set, an M×M co-occurrence matrix is ​​constructed, where M represents the total number of categories in the training set and each element C in the co-occurrence matrix ij represents the number of times the i-th category and the j-th category in the training set co-occur in the training set, i = 1, 2…M, j = 1, 2…M;

[0033] Based on the total number of times each category in the training set appears in the training set and the co-occurrence matrix, the transition probabilities of the i-th category and the j-th category in the training set are calculated respectively as elements of the co-occurrence probability transfer matrix to construct the co-occurrence probability transfer matrix.

[0034] Preferably, the calculation formula for the total number of times each category in the training set appears in the training set is:

[0035]

[0036] Based on the total number of times each category in the training set appears in the training set and the co-occurrence matrix, the transition probabilities of the i-th category and the j-th category in the training set are calculated respectively. The calculation formula is:

[0037]

[0038] Among them, T i represents the total number of times the i-th category appears in the training set, n represents the total number of categories in the training set, and C ij represents the number of times the i-th category and the j-th category co-occur in the training set, p i ' j Represents the transition probability between the i-th category and the j-th category in the training set.

[0039] Preferably, the weighted similarity matrix of the current image is constructed based on the co-occurrence probability transfer matrix, each semantic concept node and each category feature node of the current image, including:

[0040] Based on the co-occurrence probability transfer matrix, each semantic concept node and each category feature node of the current image, the weighted similarity between the αth category feature node and the i-th semantic concept node of the current image is calculated respectively as the element of the weighted similarity matrix of the current image, and the weighted similarity matrix of the current image is constructed;

[0041] The calculation formula is:

[0042]

[0043] Among them, S αi Represents the αth category feature node of the current image and the i-th semantic concept node The weighted similarity of is the i-th semantic concept node obtained by embedding the word vector of the i-th category of the training set into the feature space, represents the αth category feature node of the current image, W s and W p represents the learnable matrix, P ij represents the transition probability between the i-th category and the j-th category in the training set, N l Represents a set of semantic concept nodes, M represents the total number of categories in the training set, and K represents the total number of categories in the current image.

[0044] Preferably, the semantic-category guidance matrix of the current image is calculated based on the weighted similarity matrix of the current image, and the calculation formula is:

[0045]

[0046] in, represents the semantic-category guidance matrix of the current image, exp(.) represents the exponential function with a natural constant as the base, S αi Represents the αth category feature node of the current image and the i-th semantic concept node The weighted similarity of αj Represents the αth category feature node of the current image and the i-th semantic concept node The weighted similarity of l Represents the set of semantic concept nodes, N c Represents the category feature node set of the current image.

[0047] Preferably, the calculation formula for obtaining each enhanced category feature node of the current image is as follows:

[0048]

[0049] in, represents the αth enhanced category feature node of the current image, represents the learnable fusion parameter of semantics-category, represents the αth category feature node of the current image, represents the semantic-category guidance matrix of the current image, Represents the learnable matrix of semantics-category, P ij represents the transition probability between the i-th category and the j-th category in the training set, is the i-th semantic concept node obtained by embedding the word vector of the i-th category of the training set into the feature space, N l Represents the set of semantic concept nodes, N c Represents the set of category feature nodes of the current image, M represents the total number of categories in the training set, and K represents the total number of categories in the current image.

[0050] Preferably, the category-semantic guidance matrix of the current image is calculated based on each semantic concept node and each category feature node of the current image, and the calculation formula is:

[0051]

[0052] in, represents the semantic-category guidance matrix of the current image, exp(.) represents the exponential function with a natural constant as the base, W s and W p represents the learnable matrix, is the i-th semantic concept node obtained by embedding the word vector of the i-th category of the training set into the feature space, represents the αth category feature node of the current image, Represents the Γth category feature node of the current image, N l Represents the set of semantic concept nodes, N c Represents the set of category feature nodes of the current image, M represents the total number of categories in the training set, and K represents the total number of categories in the current image.

[0053] Preferably, the calculation formula for obtaining each enhanced semantic concept node is as follows:

[0054]

[0055] in, represents the i-th enhanced semantic concept node, is the i-th semantic concept node obtained by embedding the word vector of the i-th category of the training set into the feature space, represents the semantic-category guidance matrix of the current image, Represents the learnable fusion parameters of category-semantics, represents the αth category feature node of the current image, Represents the learnable matrix of category-semantics, N l Represents the set of semantic concept nodes, N c Represents the set of category feature nodes of the current image, M represents the total number of categories in the training set, and K represents the total number of categories in the current image.

[0056] Preferably, the training process of the dual graph structure semantic aggregation network is:

[0057] Obtain a semantic segmentation image set, and use multiple images including at least one common label as a training set;

[0058] Based on the semantic segmentation feature vector and its true label, the semantic similarity feature vector, the category feature of each category in each image and the co-occurrence feature vector of each image, a loss function is constructed, and the dual graph structure semantic aggregation network is trained using the loss function to obtain the target dual graph structure semantic aggregation network;

[0059] The loss function is constructed based on the semantic segmentation feature vector and its true label, the semantic similarity feature vector, the category feature of each category in each image and the co-occurrence feature vector of each image, including:

[0060] Based on the semantic segmentation feature vector and its true label, the cross entropy loss function is used to obtain the semantic segmentation loss;

[0061] The benchmark samples, samples of the same category as the benchmark samples, and samples of different categories from the benchmark samples are separated by semantic similarity feature vectors to extract the global features of the benchmark samples, the global features of samples of the same category as the benchmark samples, and the global features of samples of different categories from the benchmark samples;

[0062] Based on the distance between the global features of the benchmark sample and the global features of the samples of the same category as the benchmark sample, and the distance between the global features of the benchmark sample and the global features of the samples of different categories of the benchmark sample, a contrast loss function is used to obtain the global semantic similarity mining loss;

[0063] Based on the segmentation mask predicted by the category features of each category in each image and its true label, the cross entropy loss function is used to obtain the intermediate supervision loss of the intra-class dependency reasoning module;

[0064] Based on the co-occurrence feature vector of each image and the semantic relationship vector of each image obtained in the word embedding, the co-occurrence feature similarity loss is calculated using cosine similarity;

[0065] The loss function is obtained by adding the semantic segmentation loss, the weighted global semantic similarity mining loss, the intermediate supervision loss of the intra-class dependency reasoning module, the co-occurrence feature similarity loss, and the regularization loss. The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0066] The invention provides an image semantic segmentation method, which extracts the category features of each category in each image by inputting the feature map corresponding to each image into an intra-class reasoning module; the intra-class reasoning module passes the feature map corresponding to the current image through a convolution layer, obtains the decoupled representation of each category in the feature map according to the correlation of the intra-class components of the feature map corresponding to the current image, and obtains the adaptive features of each category in the current image after downsampling and upsampling the decoupled representation of each category in the current image in turn; the adaptive features of each category in the current image are passed through a self-attention mechanism to capture long-distance dependencies, and obtain the category features of each category in the current image; when processing an image, it is ensured that the processed information is unique to the category, and the confusion of representations between different categories is reduced, which helps the model to more accurately identify and classify objects and improves the accuracy of the semantic segmentation of the model.

[0067] In addition, the category features of each category in each image are input into the co-occurrence relationship reasoning module respectively, and the co-occurrence feature vector of each image is extracted. The co-occurrence relationship reasoning module uses the co-occurrence probability transfer matrix to define the probability transfer relationship between categories. The word vector of each category in the training set is obtained and embedded into the feature space as a semantic concept node, and the category features of each category in the current image are used as the category feature node of the current image; the semantic-category guidance matrix of the current image, each enhanced category feature node of the current image, the category-semantic guidance matrix of the current image, and each enhanced semantic concept node are calculated respectively through the co-occurrence probability transfer matrix, each semantic concept node and each category feature node of the current image; a double relationship is established between each enhanced semantic concept node and each enhanced category feature node of the current image. Directed connection relationships are formed between nodes to form directed edges, and a dual interactive directed graph of the current image is constructed; the dual interactive directed graph, the semantic-category guidance matrix of the current image, and the category-semantic guidance matrix of the current image are input into the graph neural network. The guidance matrix can dynamically adjust and guide the interaction of features during information transmission to ensure that nodes with strong correlation can effectively exchange information. It not only measures the similarity between different nodes, but also integrates co-occurrence relationships and contextual information to improve the expressiveness of node features. The mutual dependence between category feature nodes and semantic concept nodes is explicitly modeled through the graph neural network, and then the feature expressions of the two are interactively updated to obtain the co-occurrence feature vector of the current image, so that the model can more accurately identify small objects in the image affected by occlusion or other factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0069] Figure 1 It is the structural diagram of the dual graph structure semantic aggregation network.

[0070] Figure 2 It is a flowchart of the steps of an image semantic segmentation method of the present invention. DETAILED DESCRIPTION

[0071] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0072] Reference Figure 2 As shown, the first embodiment of the present invention provides an image semantic segmentation method, comprising the following steps:

[0073] like Figure 1 As shown, Figure 1 It is a structural diagram of a dual graph structure semantic aggregation network. Step S1: construct a dual graph structure semantic aggregation network, wherein the dual graph structure semantic aggregation network includes: an encoder, a graph neural network, an intra-class reasoning module, and a co-occurrence relationship reasoning module;

[0074] Step S2: Input multiple images including at least one common label into the encoder, and extract the feature map corresponding to each image.

[0075] The encoder is any one of VGG-16, ResNet 101, and ResNet-50.

[0076] Step S3: taking each image as a node, establishing a connection relationship between each node to form directed edges between each node, and constructing a complete directed graph based on each node and the edges between each node, wherein each node includes a self-loop edge;

[0077] Step S4: Input the complete directed graph and the feature graph corresponding to each image into the graph neural network to extract the semantic similarity feature vector;

[0078] Step S6: taking each image as a node, establishing a connection relationship between each node to form directed edges between each node, and constructing a complete directed graph G = (V, E) based on each node and the edges between each node, wherein each node includes a self-loop edge; V is the set of nodes, and E is the set of edges between nodes.

[0079] Step S7: input the complete directed graph and the feature graph corresponding to each image into the graph neural network to extract the semantic similarity feature vector;

[0080] During each training process of the network, GSSM needs to a Its adjacent node v b Generate the corresponding edge embedding. GSSM generates a dynamic weight matrix through the mapping function Among them, the mapping function f weight is any neural network, t represents the number of iterations, h a Represents node v a The feature map, h b Represents node v b The feature map of

[0081] When the mapping function f weight When it is a multi-layer perceptron (MLP), f weight The expression is:

[0082]

[0083] Among them, MLP(.) represents a multi-layer perceptron, and [.||.] represents the concatenation of features.

[0084] In order to capture the relationship between different node features and reduce the computational complexity, a bilinear interaction model is used to calculate the similarity between nodes and generate the embedded representation of the edge:

[0085]

[0086]

[0087] in, Indicates the tth iteration node v a and node v b The outgoing edge embedding of Represents the tth iteration node v b and node v a The outgoing edge embedding of represents the transposition operation, W represents the node v a The width of the feature map, H represents the node v a The height of the feature map.

[0088] For self-loop edges, the t-th iteration node v is calculated in the t-th iteration through self-attention a Self-loop edge To effectively capture non-local semantic dependencies, self-attention computes the response of a position by focusing on all positions within the same node embedding, calculated as:

[0089]

[0090] Among them, φ f (.),φ g (.),φ h (.) represents a 1×1 convolution operator.

[0091] Given a node v a, the graph convolutional neural network (GCN) collects contextual information from its neighbor nodes by applying a message function and iteratively updates v a Specifically, v a Absorb the following two types of information: ① The embedding e of self-loop edges a,a , which encodes v a Rich context-aware knowledge in a The adjacent nodes v connected to it b The edge embedding between For the former, the message function is directly from e a,a Read the message in For the latter, the message aggregation is:

[0092]

[0093] From the adjacent node v b The summarized messages and the results after the information has been propagated multiple times are:

[0094]

[0095] in, Indicates the node v in the tth iteration b To node v a The fused node embedding, softmax(.) represents the softmax activation function, Indicates the t-1th iteration node v a and node v b The outgoing edge embedding of Indicates the t-1th iteration node v b The feature map of Represents the tth iteration node v a The fused node embedding, Indicates the node v in the t-1th iteration b To node v a The fused node embedding, Indicates the node v in the t-1th iteration a The fused node embedding.

[0096] Subsequently, the node embedding is updated through the ConvGRU network. After T rounds of message passing, the final node embedding is:

[0097]

[0098] Among them, ConvGRU(.) represents the ConvGRU network, Represents the Tth iteration node v a Node embedding, Indicates the T-1th iteration node v a Node embedding, Indicates the tth iteration node v a The fused node embedding of , T represents the total number of iterations.

[0099] In the global feature readout stage, given the final node embedding First, multi-scale feature extraction is performed, the formula is:

[0100]

[0101] Among them, Downsample(.) represents the downsampling operation, s represents the reduction factor, It represents the final node embedding that is reduced by s times, and can obtain feature representations at different scales and in, represents the final node embedding that is reduced by a factor of 1, represents the final node embedding that is reduced by a factor of 2, Represents the final node embedding that is scaled down by a factor of 4.

[0102] An adaptive weighting mechanism is used to fuse multi-scale features into a global feature representation. The formula is:

[0103]

[0104] in, Represents node v a The global features of , Upsample(.) represents the upsampling operation, α s represents the weight, Indicates embedding the final node that has been reduced by s times Upsample by a factor of s to restore its size to W×H×C.

[0105] Weight α s Dynamically adjust through a trainable attention mechanism, the formula is:

[0106]

[0107] Among them, δ(.) represents a fully connected layer, which is used to generate weight distribution.

[0108] Final global feature representation is input into a fully connected layer, outputting the semantic similarity feature vector F g .

[0109] Step S5: Input the feature map corresponding to each image into the intra-class reasoning module to extract the category features of each category in each image, including:

[0110] Pass the feature map corresponding to the current image through the convolution layer to obtain the decoupled representation of each category in the current image;

[0111] The decoupled representation of each category in the current image is downsampled and upsampled in turn to obtain the adaptive features of each category in the current image;

[0112] The adaptive features of each category in the current image are used through the self-attention mechanism to capture long-distance dependencies and obtain the category features of each category in the current image;

[0113] In this embodiment, preferably, the decoupled representation of the αth category of the current image is sequentially downsampled and upsampled to obtain the adaptive feature of the αth category of the current image, and the calculation formula is:

[0114]

[0115]

[0116] in, represents the decoupled representation of the αth category of the current image after downsampling, Represents the adaptive features of the αth category of the current image, Downsample(.) represents the downsampling operation, and Upsample(.) represents the upsampling operation.

[0117] In order to ensure that the adaptive features of each category in the current image only contain the information of that category, an intermediate supervision mechanism is introduced to help the model learn gradually during the training process and ensure that the representation of each category in the current image does not confuse the features of other categories.

[0118] Use the adaptive features of each category in the current image to perform an intermediate segmentation task. Based on the predicted segmentation mask of each category in the current image and the true segmentation label of the category, calculate the intermediate supervision loss of the intra-class dependency reasoning module. The formula is as follows:

[0119]

[0120] in, represents the intermediate supervision loss of the αth category of the current image, represents the segmentation mask of the αth category prediction of the current image. pixels, represents the true segmentation label of the αth category of the current image. pixels, and N represents the total number of pixels of the segmentation mask predicted for the αth category of the current image.

[0121] The adaptive features of each category in the current image are used through the self-attention mechanism to capture long-distance dependencies and obtain the category features of each category in the current image;

[0122] After obtaining the adaptive features for each category, how to effectively capture the relationship between different positions in the feature map is especially important for objects with complex internal structures. For example, a face consists of eyes, nose, and mouth, while a car consists of wheels, body, and windows.

[0123] Traditional convolution operations have limitations in capturing long-distance dependencies because they can only process features in local areas. The self-attention mechanism can effectively capture long-distance dependencies. By calculating the dot product of the query vector Q, the key vector K, and the value vector V, associations can be established between different positions of the adaptive features. Through the attention mechanism, the category features of each category in the current image are obtained.

[0124] The adaptive features of each category in the current image are captured through the self-attention mechanism to capture long-distance dependencies, and the category features of each category in the current image are obtained. The calculation formula is:

[0125] X' α =Attention(Q,K,V),

[0126] Among them, X' α is the category feature of the αth category of the current image, Attention(.) is the self-attention mechanism, Q represents the query vector of the αth category of the current image, K represents the key vector of the αth category of the current image, V represents the value vector of the αth category of the current image, and the query vector Q, key vector K and value vector V are the adaptive features of the αth category of the current image multiplied by three different linear projection layers respectively.

[0127] The intra-class reasoning module decouples the representations of different categories so that the representation of each category only contains spatial or semantic information related to one category. However, in the real world, in addition to the spatial structure or contextual relationship within the object, the dependencies between visual entities are ubiquitous. These visual concepts and the relationships between them provide prior knowledge about the real world. With this prior knowledge, it is not only possible to handle the uncertainty of fuzzy categories, but also to accurately identify small objects in the image that are affected by occlusion or other factors. To this end, a semantically associated co-occurrence relationship reasoning module is designed to generate co-occurrence feature vectors, and the knowledge at the dataset level is introduced to guide the image to learn contextual information.

[0128] Step S6: inputting the category features of each category in each image into the co-occurrence relationship reasoning module to extract the co-occurrence feature vector of each image;

[0129] In this embodiment, preferably, the step of inputting the category features of each category in each image into the co-occurrence relationship reasoning module to extract the co-occurrence feature vector of each image includes:

[0130] Get the word vector of each category in the training set, and embed the word vector of each category in the training set into the feature space as a semantic concept node;

[0131] Based on the total number of times each category in the training set appears in the training set and the number of times each category co-occurs with other categories in the training set, a co-occurrence probability transfer matrix is ​​constructed;

[0132] In this embodiment, preferably, the co-occurrence probability transfer matrix is ​​constructed based on the total number of times each category in the training set appears in the training set and the number of times each category co-occurs with other categories in the training set, including:

[0133] Based on the total number of times each category appears in the training set and the number of times each category co-occurs with other categories in the training set, an M×M co-occurrence matrix is ​​constructed, where M represents the total number of categories in the training set and each element C in the co-occurrence matrix ij represents the number of times the i-th category and the j-th category in the training set co-occur in the training set, i = 1, 2…M, j = 1, 2…M;

[0134] Based on the total number of times each category in the training set appears in the training set and the co-occurrence matrix, the transition probabilities of the i-th category and the j-th category in the training set are calculated respectively as elements of the co-occurrence probability transfer matrix to construct the co-occurrence probability transfer matrix.

[0135] The total number of times each category in the training set appears in the training set is calculated as:

[0136]

[0137] In order to exclude the co-occurrence cases with a small probability, only the cases where the number of co-occurrence categories in the training set is less than or equal to 3 are counted. Based on the total number of times each category appears in the training set and the co-occurrence matrix, the transition probabilities of the i-th category and the j-th category in the training set are calculated respectively. The calculation formula is:

[0138]

[0139] Among them, T i represents the total number of times the i-th category appears in the training set, n represents the total number of categories in the training set, and C ijrepresents the number of times the i-th category and the j-th category co-occur in the training set, p i ' j Represents the transition probability between the i-th category and the j-th category in the training set.

[0140] For each image, the category features of each category in the current image are used as the category feature nodes of the current image;

[0141] Based on the co-occurrence probability transfer matrix, each semantic concept node and each category feature node of the current image, a weighted similarity matrix of the current image is constructed;

[0142] In this embodiment, preferably, the weighted similarity matrix of the current image is constructed based on the co-occurrence probability transfer matrix, each semantic concept node and each category feature node of the current image, including:

[0143] Based on the co-occurrence probability transfer matrix, each semantic concept node and each category feature node of the current image, the weighted similarity between the αth category feature node and the i-th semantic concept node of the current image is calculated respectively as the element of the weighted similarity matrix of the current image, and the weighted similarity matrix of the current image is constructed;

[0144] The calculation formula is:

[0145]

[0146] Among them, S αi Represents the αth category feature node of the current image and the i-th semantic concept node The weighted similarity of is the i-th semantic concept node obtained by embedding the word vector of the i-th category of the training set into the feature space, represents the αth category feature node of the current image, W s and W p represents the learnable matrix, P ij represents the transition probability between the i-th category and the j-th category in the training set, N l Represents a set of semantic concept nodes, M represents the total number of categories in the training set, and K represents the total number of categories in the current image.

[0147] Based on the weighted similarity matrix of the current image, the semantic-category guidance matrix of the current image is calculated;

[0148] The calculation formula is:

[0149]

[0150] in, represents the semantic-category guidance matrix of the current image, exp(.) represents the exponential function with a natural constant as the base, S αi Represents the αth category feature node of the current image and the i-th semantic concept node The weighted similarity of αj Represents the αth category feature node of the current image and the i-th semantic concept node The weighted similarity of l Represents the set of semantic concept nodes, N c Represents the category feature node set of the current image.

[0151] According to the semantic-category guidance matrix of the current image and each semantic concept node, each enhanced category feature node of the current image is calculated;

[0152] The calculation formula is:

[0153]

[0154] in, represents the αth enhanced category feature node of the current image, represents the learnable fusion parameter of semantics-category, represents the αth category feature node of the current image, represents the semantic-category guidance matrix of the current image, Represents the learnable matrix of semantics-category, P ij represents the transition probability between the i-th category and the j-th category in the training set, is the i-th semantic concept node obtained by embedding the word vector of the i-th category of the training set into the feature space, N l Represents the set of semantic concept nodes, N c Represents the set of category feature nodes of the current image, M represents the total number of categories in the training set, and K represents the total number of categories in the current image.

[0155] Based on each semantic concept node and each category feature node of the current image, the category-semantic guidance matrix of the current image is calculated;

[0156] The calculation formula is:

[0157]

[0158] in, represents the semantic-category guidance matrix of the current image, exp(.) represents the exponential function with a natural constant as the base, W s and W p represents the learnable matrix, is the i-th semantic concept node obtained by embedding the word vector of the i-th category of the training set into the feature space, represents the αth category feature node of the current image, Represents the Γth category feature node of the current image, N l Represents the set of semantic concept nodes, N c Represents the set of category feature nodes of the current image, M represents the total number of categories in the training set, and K represents the total number of categories in the current image.

[0159] According to the category-semantic guidance matrix of the current image and each category feature node of the current image, each enhanced semantic concept node is calculated;

[0160] The calculation formula is:

[0161]

[0162] in, represents the i-th enhanced semantic concept node, is the i-th semantic concept node obtained by embedding the word vector of the i-th category of the training set into the feature space, represents the semantic-category guidance matrix of the current image, Represents the learnable fusion parameters of category-semantics, represents the αth category feature node of the current image, Represents the learnable matrix of category-semantics, N l Represents the set of semantic concept nodes, N c Represents the set of category feature nodes of the current image, M represents the total number of categories in the training set, and K represents the total number of categories in the current image.

[0163] The transfer of information between semantic concept nodes and category feature nodes is achieved through the Guidance Matrix. The role of the Guidance Matrix is ​​to dynamically adjust and guide the interaction of features during the information transfer process to ensure that nodes with strong correlation can effectively exchange information. It not only measures the similarity between different nodes, but also integrates co-occurrence relationships and contextual information to improve the expressiveness of node features.

[0164] Based on the visual co-occurrence information provided by the category feature nodes, the information of the semantic concept nodes can be increased or strengthened to ensure that the semantic representation is more consistent with the observed data in the actual image. This helps the model capture and understand the co-occurrence patterns that are not fully reflected in the original dataset annotations, so that the semantic graph can more accurately reflect the semantic structure of the image content. l and the category feature node set N c ,in,

[0165] Establish a bidirectional connection relationship between each enhanced semantic concept node and each enhanced category feature node of the current image to form a directed edge between each node, and construct a dual interactive directed graph of the current image based on each enhanced semantic concept node, each enhanced category feature node of the current image and the edges between each node;

[0166] The dual interaction directed graph, semantic-category guidance matrix and category-semantic guidance matrix of the current image are input into the graph neural network to obtain the co-occurrence feature vector of the current image.

[0167] The interdependence between category feature nodes and semantic concept nodes is explicitly modeled through graph neural networks, and the feature expressions of both are then updated interactively.

[0168] Graph neural networks use graph neural networks to carry out message passing and iterative reasoning between multiple images, learn high-order relationships between multiple images, collect image-level contextual knowledge, and extract common semantic representations of this group of images.

[0169] For the category feature node set For each node in , and obtain other semantic nodes that have a significant co-occurrence relationship with the semantic node. Then, based on the information in the co-occurrence matrix, the node features of these co-occurring objects in the visual graph are enhanced. This step means that if "dog" and "ball" have a high frequency of co-occurrence in the co-occurrence matrix, then when processing images containing "dog", we will pay special attention to and enhance the feature representation of "ball" to achieve feature enhancement of the category feature node about the "ball" node.

[0170] When a new object combination is identified in the category feature node, this information can be fed back to the co-occurrence matrix and the features of the semantic concept node can be updated accordingly. This means that if "dog" and "beach" are newly found to appear together frequently in a large number of images, this new co-occurrence pattern will be added to the co-occurrence matrix and the semantic concept node. Based on the new object combination information, the co-occurrence matrix is ​​updated with the formula:

[0171]

[0172] ΔC ij represents the increment of the number of co-occurrences between the i-th category and the j-th category in the training set, ΔT i Represents the increase in the total number of times the i-th category appears in the training set.

[0173] The interdependence between category feature nodes and semantic concept nodes is explicitly modeled through graph neural networks, and then the feature expressions of both are updated interactively. Specifically, each semantic concept node, each category feature node of the current image, and the two guidance matrices of the interaction process are updated. and As input, the semantic concept node and the category feature node of the current image are updated as follows:

[0174]

[0175] Among them, S t represents the category node feature of the current image after the tth iteration, F t represents the semantic node features after the tth iteration, W t represents the learnable transformation matrix after the tth iteration, σ(·) represents the relu activation function, S (t+1) represents the category node feature of the current image after the t+1th iteration, F (t+1) Represents the semantic node features after the t+1th iteration.

[0176] The category node features and semantic node features of the current image after the t+1th iteration are fused to obtain ω t+1 =S t+1 ∪F t+1 , where ω t+1 Represents the fused feature vector after the t+1th iteration.

[0177] Then use a learnable aggregation function to average pool the pairs ω t+1 Aggregate to get the final co-occurrence feature vector

[0178] in, represents the final co-occurrence feature vector, It can be further transformed through a fully connected layer to obtain the co-occurrence feature vector F of the current image co .

[0179]

[0180] Among them, W out and b out are the weights and biases of the fully connected layers.

[0181] Step S8: After respectively concatenating the semantic similarity feature vector with the co-occurrence feature vector of each image, a concatenated feature vector of each image is obtained;

[0182] The semantic information enhanced feature representation module fuses the semantic similarity feature vector obtained by the semantic similarity mining module and the co-occurrence feature vector obtained by the co-occurrence relationship reasoning module, thereby enhancing the feature representation capability.

[0183] Step S8: The concatenated feature vector of each image is sequentially passed through a fully connected layer and a Softmax activation function to calculate the adaptive weight of each image;

[0184] Assume that the parameters of the fully connected layer are 2 represents the width of the fully connected layer, d represents the height of the fully connected layer, and the calculation formula of the adaptive weight is:

[0185] [λ 1 ,λ 2 ]=softmax(W w [F g ; F co ]+b w ,

[0186] Among them, W w represents the weight of the fully connected layer, b w represents the bias of the fully connected layer, F g represents the semantic similarity feature vector, F co represents the co-occurrence feature vector of the current image, [F g ; F co ] represents the concatenated feature vector of the current image, λ 1 and λ 2 is an adaptive weight that satisfies λ 1 +λ 2 =1.

[0187] Step S9: performing weighted calculation on the spliced ​​feature vector of each image according to the adaptive weight of each image, passing the weighted feature vector of each image through the fully connected layer, and outputting the semantic segmentation feature vector of each image;

[0188] The calculation formula is:

[0189] F weighted =λ 1 F g +λ 2 F co ,

[0190] Among them, F weighted Represents the weighted feature vector of the current image.

[0191] Step S10: The semantic segmentation feature vector of each image is passed through the classifier, the Softmax activation function, and the argmax operation in sequence to obtain a semantic segmentation map of each image.

[0192] In this embodiment, preferably, the training process of the dual graph structure semantic aggregation network is:

[0193] Obtain a semantic segmentation image set, and use multiple images including at least one common label as a training set;

[0194] Based on the semantic segmentation feature vector and its true label, the semantic similarity feature vector, the category feature of each category in each image and the co-occurrence feature vector of each image, a loss function is constructed, and the dual graph structure semantic aggregation network is trained using the loss function to obtain the target dual graph structure semantic aggregation network;

[0195] set up is a labeled training set, is the εth image in the training set, is the label corresponding to the εth image in the training set, N s is the total number of samples in the training set, W is the width of the εth image in the training set, H is the height of the εth image in the training set, and C is the number of channels of the εth image in the training set.

[0196] The loss function is constructed based on the semantic segmentation feature vector and its true label, the semantic similarity feature vector, the category feature of each category in each image and the co-occurrence feature vector of each image, including:

[0197] The entire network is trained in an end-to-end manner with multi-granularity supervision, including:

[0198] It is used to measure the difference between the semantic segmentation feature vector and the true label. Based on the semantic segmentation feature vector and its true label, the cross entropy loss function is used to obtain the semantic segmentation loss. The calculation formula is:

[0199]

[0200] Among them, L seg is the semantic segmentation loss, N x is the total number of pixels of the semantic segmentation feature vector, L C is the total number of categories, y u,l is the true label of the u-th pixel (one-hot encoding), P u,ρ represents the predicted probability that the u-th pixel belongs to category ρ.

[0201] It is used to ensure that the model can effectively extract semantic similarity feature vectors between images, and extract global features of benchmark samples, global features of samples of the same category as benchmark samples, and global features of samples of different categories from benchmark samples through semantic similarity feature vectors;

[0202] Based on the distance between the global features of the benchmark sample and the global features of the samples of the same category as the benchmark sample, and the distance between the global features of the benchmark sample and the global features of the samples of different categories of the benchmark sample, the contrast loss function is used to obtain the global semantic similarity mining loss. The calculation formula is:

[0203]

[0204] Among them, L GSSM represents the global semantic similarity mining loss, represents the number of sample pairs, represents the global feature of the θth sample, Representation and The global features of positive samples of the same category, Representation and The global features of negative samples of different categories, δ 1 is the preset boundary threshold.

[0205] Based on the segmentation mask predicted by the category features of each category in each image and its true label, the cross entropy loss function is used to obtain the intermediate supervision loss of the intra-class dependency reasoning module;

[0206] The formula is as follows:

[0207]

[0208] in, represents the intermediate supervision loss of the αth category of the current image, Y α ' ,ζ represents the segmentation mask of the αth category prediction of the current image. pixels, Y α,ζ represents the ζ-th pixel of the true segmentation label of the α-th category of the current image, and N represents the total number of pixels of the segmentation mask predicted for the α-th category of the current image.

[0209] Based on the co-occurrence feature vector of each image and the semantic relationship vector of each image obtained in the word embedding, the co-occurrence feature similarity loss is calculated using cosine similarity; it is used to ensure that the co-occurrence feature vector generated by the model is consistent with Sim ee′ The similarity between . It is defined as follows:

[0210]

[0211] Among them, L co_sim represents the co-occurrence feature similarity loss, ∈ represents the preset similarity threshold, represents the co-occurrence feature vector of the εth image, Sim ε Represents the semantic relationship vector between each category of the ε-th image obtained from word embedding.

[0212] The loss function is obtained by adding the semantic segmentation loss, the weighted global semantic similarity mining loss, the intermediate supervision loss of the intra-class dependency reasoning module, the co-occurrence feature similarity loss, and the regularization loss.

[0213] In this embodiment, L2 regularization is used to define the regularization loss to prevent the model from overfitting.

[0214] This embodiment designs the following objective function:

[0215]

[0216] in, represents the objective function, L seg represents the semantic segmentation loss, L inter represents the intermediate supervision loss of the intra-class dependency reasoning module, L co_sim represents the co-occurrence feature similarity loss, L reg (Θ) represents the regularization loss, Θ represents the dual graph structure semantic aggregation network, α′, β′, and γ′ are and and L co_sim The weight hyperparameter of , λ is the weight hyperparameter of the regularization term

[0217] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0218] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0219] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0220] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0221] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A method for image semantic segmentation, characterized in that: The following steps are involved: Constructing a dual graph structure semantic aggregation network, wherein the dual graph structure semantic aggregation network includes: an encoder, a graph neural network, an intra-class reasoning module, and a co-occurrence relationship reasoning module; Input multiple images including at least one common label into the encoder, and extract the feature map corresponding to each image; Each image is taken as a node, and a connection relationship is established between each node to form a directed edge between each node. Based on each node and the edge between each node, a complete directed graph is constructed, in which each node includes a self-loop edge; The fully directed graph and the feature map corresponding to each image are input into the graph neural network to extract the semantic similarity feature vector; The feature map corresponding to each image is input into the intra-class reasoning module to extract the category features of each category in each image, including: Pass the feature map corresponding to the current image through the convolution layer to obtain the decoupled representation of each category in the current image; The decoupled representation of each category in the current image is downsampled and upsampled in turn to obtain an adaptive feature representation of each category in the current image; The adaptive feature representation of each category in the current image is used through the self-attention mechanism to capture long-distance dependencies and obtain the category features of each category in the current image; The category features of each category in each image are input into the co-occurrence relationship reasoning module to extract the co-occurrence feature vector of each image; After concatenating the semantic similarity feature vector with the co-occurrence feature vector of each image, a concatenated feature vector of each image is obtained; The concatenated feature vector of each image is passed through the fully connected layer and the Softmax activation function in turn to calculate the adaptive weight of each image; The concatenated feature vector of each image is weighted according to the adaptive weight of each image, and the weighted feature vector of each image is passed through the fully connected layer to output the semantic segmentation feature vector of each image; The semantic segmentation feature vector of each image is passed through the classifier, Softmax activation function, and argmax operation in turn to obtain the semantic segmentation map of each image.

2. The image semantic segmentation method according to claim 1, characterized in that: The step of inputting the category features of each category in each image into the co-occurrence relationship reasoning module to extract the co-occurrence feature vector of each image includes: Get the word vector of each category in the training set, and embed the word vector of each category in the training set into the feature space as a semantic concept node; Based on the total number of times each category in the training set appears in the training set and the number of times each category co-occurs with other categories in the training set, a co-occurrence probability transfer matrix is ​​constructed; For each image, the category features of each category in the current image are used as the category feature nodes of the current image; Based on the co-occurrence probability transfer matrix, each semantic concept node and each category feature node of the current image, a weighted similarity matrix of the current image is constructed; Based on the weighted similarity matrix of the current image, the semantic-category guidance matrix of the current image is calculated; According to the semantic-category guidance matrix of the current image and each semantic concept node, each enhanced category feature node of the current image is calculated; Based on each semantic concept node and each category feature node of the current image, the category-semantic guidance matrix of the current image is calculated; According to the category-semantic guidance matrix of the current image and each category feature node of the current image, each enhanced semantic concept node is calculated; Establish a bidirectional connection relationship between each enhanced semantic concept node and each enhanced category feature node of the current image to form a directed edge between each node, and construct a dual interactive directed graph of the current image based on each enhanced semantic concept node, each enhanced category feature node of the current image and the edges between each node; The dual interaction directed graph, semantic-category guidance matrix and category-semantic guidance matrix of the current image are input into the graph neural network to obtain the co-occurrence feature vector of the current image.

3. The image semantic segmentation method according to claim 2, characterized in that: The co-occurrence probability transfer matrix is ​​constructed based on the total number of times each category in the training set appears in the training set and the number of times each category co-occurs with other categories in the training set, including: Based on the total number of times each category appears in the training set and the number of times each category co-occurs with other categories in the training set, an M×M co-occurrence matrix is ​​constructed, where M represents the total number of categories in the training set and each element C in the co-occurrence matrix ij represents the number of times the i-th category and the j-th category in the training set co-occur in the training set, i = 1, 2…M, j = 1, 2…M; Based on the total number of times each category in the training set appears in the training set and the co-occurrence matrix, the transition probabilities of the i-th category and the j-th category in the training set are calculated respectively as elements of the co-occurrence probability transfer matrix to construct the co-occurrence probability transfer matrix.

4. The image semantic segmentation method according to claim 3, characterized in that: The formula for calculating the total number of times each category appears in the training set is: Based on the total number of times each category appears in the training set and the co-occurrence matrix, the transition probabilities of the i-th category and the j-th category in the training set are calculated respectively. The calculation formula is: Among them, T i represents the total number of times the i-th category appears in the training set, n represents the total number of categories in the training set, and C ij represents the number of times the i-th category and the j-th category co-occur in the training set, p i ′ j Represents the transition probability between the i-th category and the j-th category in the training set.

5. The image semantic segmentation method according to claim 4, characterized in that: The weighted similarity matrix of the current image is constructed based on the co-occurrence probability transfer matrix, each semantic concept node and each category feature node of the current image, including: Based on the co-occurrence probability transfer matrix, each semantic concept node and each category feature node of the current image, the weighted similarity between the αth category feature node and the i-th semantic concept node of the current image is calculated respectively as the element of the weighted similarity matrix of the current image, and the weighted similarity matrix of the current image is constructed; The calculation formula is: Among them, S αi Represents the αth category feature node of the current image and the i-th semantic concept node The weighted similarity of is the i-th semantic concept node obtained by embedding the word vector of the i-th category of the training set into the feature space, represents the αth category feature node of the current image, W s and W p represents the learnable matrix, P ij represents the transition probability between the i-th category and the j-th category in the training set, N l Represents a set of semantic concept nodes, M represents the total number of categories in the training set, and K represents the total number of categories in the current image.

6. The image semantic segmentation method according to claim 5, characterized in that: The weighted similarity matrix based on the current image is used to calculate the semantic-category guidance matrix of the current image. The calculation formula is: in, represents the semantic-category guidance matrix of the current image, exp(.) represents the exponential function with a natural constant as the base, S αi Represents the αth category feature node of the current image and the i-th semantic concept node The weighted similarity of αj Represents the αth category feature node of the current image and the i-th semantic concept node The weighted similarity of l Represents the set of semantic concept nodes, N c Represents the category feature node set of the current image.

7. The image semantic segmentation method according to claim 6, characterized in that: The calculation formula for obtaining each enhanced category feature node of the current image is as follows: in, represents the αth enhanced category feature node of the current image, represents the learnable fusion parameter of semantics-category, represents the αth category feature node of the current image, represents the semantic-category guidance matrix of the current image, Represents the learnable matrix of semantics-category, P ij represents the transition probability between the i-th category and the j-th category in the training set, is the i-th semantic concept node obtained by embedding the word vector of the i-th category of the training set into the feature space, N l Represents the set of semantic concept nodes, N c Represents the set of category feature nodes of the current image, M represents the total number of categories in the training set, and K represents the total number of categories in the current image.

8. The image semantic segmentation method according to claim 2, characterized in that: Based on each semantic concept node and each category feature node of the current image, the category-semantic guidance matrix of the current image is calculated, and the calculation formula is: in, represents the semantic-category guidance matrix of the current image, exp(.) represents the exponential function with a natural constant as the base, W s and W p represents the learnable matrix, is the i-th semantic concept node obtained by embedding the word vector of the i-th category of the training set into the feature space, represents the αth category feature node of the current image, Represents the Γth category feature node of the current image, N l Represents the set of semantic concept nodes, N c Represents the set of category feature nodes of the current image, M represents the total number of categories in the training set, and K represents the total number of categories in the current image.

9. The image semantic segmentation method according to claim 8, characterized in that: The calculation formula for obtaining each enhanced semantic concept node is as follows: in, represents the i-th enhanced semantic concept node, is the i-th semantic concept node obtained by embedding the word vector of the i-th category of the training set into the feature space, represents the semantic-category guidance matrix of the current image, Represents the learnable fusion parameters of category-semantics, represents the αth category feature node of the current image, Represents the learnable matrix of category-semantics, N l Represents the set of semantic concept nodes, N c Represents the set of category feature nodes of the current image, M represents the total number of categories in the training set, and K represents the total number of categories in the current image.

10. The image semantic segmentation method according to claim 1, characterized in that: The training process of the dual graph structure semantic aggregation network is: Obtain a semantic segmentation image set, and use multiple images including at least one common label as a training set; Based on the semantic segmentation feature vector and its true label, the semantic similarity feature vector, the category feature of each category in each image and the co-occurrence feature vector of each image, a loss function is constructed, and the dual graph structure semantic aggregation network is trained using the loss function to obtain the target dual graph structure semantic aggregation network; The loss function is constructed based on the semantic segmentation feature vector and its true label, the semantic similarity feature vector, the category feature of each category in each image and the co-occurrence feature vector of each image, including: Based on the semantic segmentation feature vector and its true label, the cross entropy loss function is used to obtain the semantic segmentation loss; The benchmark samples, samples of the same category as the benchmark samples, and samples of different categories from the benchmark samples are separated by semantic similarity feature vectors to extract the global features of the benchmark samples, the global features of samples of the same category as the benchmark samples, and the global features of samples of different categories from the benchmark samples; Based on the distance between the global features of the benchmark sample and the global features of the samples of the same category as the benchmark sample, and the distance between the global features of the benchmark sample and the global features of the samples of different categories of the benchmark sample, a contrast loss function is used to obtain the global semantic similarity mining loss; Based on the segmentation mask predicted by the category features of each category in each image and its true label, the cross entropy loss function is used to obtain the intermediate supervision loss of the intra-class dependency reasoning module; Based on the co-occurrence feature vector of each image and the semantic relationship vector of each image obtained in the word embedding, the co-occurrence feature similarity loss is calculated using cosine similarity; The loss function is obtained by adding the semantic segmentation loss, the weighted global semantic similarity mining loss, the intermediate supervision loss of the intra-class dependency reasoning module, the co-occurrence feature similarity loss, and the regularization loss.

Images