Semantic-space-mapping-based disparity map processing method for zero-sample disparity map embedding

Abstract

The invention relates to a semantic space mapping-based zero sample disparity map embedding disparity map processing method, which comprises the following steps of: in an off-line stage, constructing a RESCN (Response Error Session Control Network) containing an adaptive simple map convolutional layer, and training a preprocessed data set serving as a sample; in the online stage, the trained RESCN is adopted to process different illustration image information received in real time; according to the method, through adaptive convolution, feature information of multi-hop neighbors in the image is effectively captured, and embedded representation of the different matching image is optimized. And secondly, class semantic description and a graph structure representation loss function are introduced, and the processing capability of the network on multi-label and unbalanced data is enhanced, so that the performance of the model in a different illustration graph zero sample learning task is remarkably improved, and good performance in a same illustration graph scene is kept.

Description

Technical Field

[0001] The present invention relates to a technology in the field of graph neural networks, specifically a heterogeneous graph processing method based on zero-sample heterogeneous graph embedding of semantic space mapping. Background Art

[0002] Most existing algorithms in the field of heterogeneous graph embedding focus on node features and ignore graph structure information, while algorithms that combine graph structure information rarely consider heterogeneous graph scenarios. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art and the problem that the algorithm designed based on heterogeneous graphs cannot handle unbalanced labels, the present invention proposes a heterogeneous graph processing method based on zero-shot heterogeneous graph embedding of semantic space mapping, which effectively captures the feature information of multi-hop neighbors in the graph through adaptive convolution and optimizes the embedding representation of heterogeneous graphs. Secondly, the semantic description and graph structure representation loss function are introduced to enhance the network's processing ability for multi-label and unbalanced data, thereby significantly improving the performance of the model in the zero-shot learning task of heterogeneous graphs and maintaining good performance in the same-matching graph scenario.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a heterogeneous graph processing method based on zero-shot heterogeneous graph embedding of semantic space mapping. In the offline stage, a replanned simplified graph convolutional network (RESCN) including an adaptive simple graph convolution (ASGC) layer is constructed, and a preprocessed data set is used as a sample to train the network. In the online stage, the trained RESCN is used to process heterogeneous graph information received in real time.

[0006] The method specifically comprises the following steps:

[0007] Step 1) Construct an adaptive simple graph convolution layer, set corresponding filters for each different feature, and adapt to the homogeneous and heterogeneous structures by changing the weights of different filters. For each node's feature, the feature is obtained by ASGC convolution, and the homogeneous graph and heterogeneous graph are processed at the same time. Similar features are adaptively convolved, and the smoothed features will be almost consistent with SGC. For neighbor nodes with dissimilar heterogeneous graph features, their features will be filtered to avoid the noise caused by the aggregation features of heterogeneous structures.

[0008] Step 2) Construct a RESCN including an SVD singular value decomposition layer, an ASGC layer, and a linear layer, wherein: the SVD singular value decomposition layer performs SVD singular value decomposition on the feature matrix in the read feature file to obtain a full rank decomposition of the feature matrix. The ASGC layer aggregates the feature vectors of the graph nodes in an adaptive manner and obtains the aggregated features by calculating the correlation with the features of neighboring nodes. The linear layer projects the aggregated features into the semantic space to obtain a graph embedding for the classification task.

[0009] Step 3) Train the constructed RESCN by calculating the semantic description loss function between the predicted semantic description vector of the visible class and the actual class semantic description vector to capture the class semantic information of the graph, and calculate the graph structure representation loss function to capture the information of the graph structure. The node sequence is obtained by random depth walk on the graph structure, and the skip-gram model is used to calculate the maximum likelihood estimate of the node sequence. The two loss functions are combined to jointly train the model and optimize the model parameters at the semantic level and the graph structure level.

[0010] Step 4) Experimentally verify the model performance in the online stage, including selecting experimental datasets, randomly sampling training sets and test sets, using graph structure information and completely unbalanced datasets to obtain node embeddings, training linear SVM classifiers for node classification, and evaluating method performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of the present invention;

[0012] Figure 2 Schematic diagram of RESCN according to the embodiment. DETAILED DESCRIPTION

[0013] like Figure 1 As shown, this embodiment relates to a heterogeneous processing method for zero-sample heterogeneous graph embedding based on semantic space mapping, including:

[0014] Step 1) construct an adaptive simple graph convolutional network, specifically including:

[0015] 1.1) Set corresponding filters for each different feature, and adapt to the same matching and different matching structures by changing the weights of different filters. Specifically, for each node feature x, the feature x is obtained by ASGC convolution Among them: S is the adjacency matrix after Laplace normalization, K is the hyperparameter that controls the width of the ASGC convolution, and c k is the scaling factor of the node feature x estimated linearly using the k-th step feature aggregation, that is: x = (S 1 x;S 2 x;...;S k x)c,x SGCH =(S 1x;S 2 x;...;S k x)c.

[0016] 1.2) Process the same-matching graph and the different-matching graph at the same time, and adaptively convolve similar features. Specifically, when it is a same-matching graph, since the neighbor features of the nodes are almost the same, the smoothed features will be almost consistent with SGC; when it is a different-matching graph, for neighbor nodes with different node features, their features will be filtered, thereby effectively avoiding the noise caused by the aggregation features of the different-matching structure. The filter obtained in step 1.1 is adapted to the feature convolution of the same-matching and different-matching structures by adjusting the weights in this step. The features processed in this step will be further aggregated and projected through the ASGC layer of RESCN in step 2.

[0017] Step 2) Build Figure 2 The RESCN shown in the figure includes: an SVD singular value decomposition layer, an ASGC layer and a linear layer, wherein: the SVD singular value decomposition layer performs SVD singular value decomposition on the feature matrix in the read feature file to obtain a full rank decomposition of the feature matrix; the ASGC layer aggregates the feature vectors of the graph nodes in an adaptive manner, and obtains the aggregated features by calculating the correlation with the features of neighboring nodes; the linear layer projects the aggregated features into the semantic space to obtain a graph embedding representation for the classification task.

[0018] Step 3) Training RESCN, including:

[0019] 3.1) Input multiple datasets as training samples into RESCN to obtain the semantic description vector of the predicted visible class;

[0020] 3.2) By calculating the semantic description loss function between the predicted semantic description vector of the visible class and the actual class semantic description vector in: is the class semantic prediction vector of the i-th node in the semantic space S, is the actual class semantic description vector of the i-th node, which is read by the function get: The feature vector of the node is converted to obtain i is the feature vector of the ith node. Through learning at the class semantic level, the present invention can capture the class semantic information of the graph, so that the learned graph embedding can reflect the supervision information of the visible class and the invisible class.

[0021] 3.3) Calculate the graph structure representation loss function to capture the information of the graph structure, specifically: Where: M = (A + A * A) / 2, A is the adjacency matrix of the heterogeneous graph, UU' is the embedding obtained by the model, and loss() is the loss function.

[0022] 3.4) Learn the topological structure of the graph by randomly walking the graph structure to obtain the node sequence, and convert the obtained node sequence w = {v i ...v w}∈Ω is regarded as a word sequence, and the skip-gram model is used to calculate the maximum likelihood estimate of the node sequence Where: r is the range of the depth walk window, the probability Pr(v j |v i ) represents the slave node v i Random walk to node v j The probability of using the softmax function is calculated The deep walk is to perform matrix decomposition on the matrix M, where in: is the adjacency matrix after row normalization, e i is the indicator vector, the i-th item is 1 and the rest are 0.

[0023] In this embodiment, t=2.

[0024] The maximum likelihood estimation is used to capture the topological structure information of the graph and generate node representations. These representations are combined with the graph structure representation loss function to jointly optimize the model.

[0025] 3.5) After training the model through the semantic description loss function, a representation U with semantic information is generated (1) , retrain the model through the graph structure representation loss function to generate a representation U with graph structure information (2) After that, the two representations are concatenated and normalized to obtain the final node embedding U = Normalize ([U (1) ,U (2) ]), where Normalize(.) is the normalization function.

[0026] The regularization operation concatenates and regularizes the semantic information representation and the graph structure information representation to obtain the final node embedding. Therefore, the maximum likelihood estimation plays a bridging role in the whole process, organically combining the graph topology information with the semantic information and the graph structure information to optimize the model representation.

[0027] Preferably, the semantic description loss function and the graph structure representation loss function are combined, and training is performed simultaneously at the semantic level and the graph structure level to optimize the model parameters. Specifically, in each training iteration, the semantic description loss and the graph structure representation loss are calculated, and the weighted sum of the two is used as the total loss function. The model parameters are updated through the back-propagation algorithm to optimize the model at two levels simultaneously.

[0028] Step 4) Online experiments verify model performance, including:

[0029] 4.1) Select the experimental data set shown in Table 1:

[0030] Table 1

[0031] 4.2) Randomly sample a portion of nodes as a training set The remaining nodes are used as the test set; nodes with certain class labels are selected as unseen classes, and nodes with such labels are deleted from the training set to form a completely unbalanced data set. Among them: the percentage of nodes in the training set is 30%.

[0032] For simplicity, the first two categories among all categories are directly selected as unseen categories and deleted from the training set to construct a completely unbalanced scenario, in which the number of deleted nodes accounts for about 20% of the training set.

[0033] 4.3) Using graph structure information and completely unbalanced datasets After getting the node embedding U, based on U and A linear SVM classifier was trained and used on the test set to classify nodes to evaluate the performance of various methods.

[0034] In the ASGC layer, the number of iterations of the least squares method is set to 1000, the number of ASGC layers K is set to 2, and the dimension of the hidden layer of the graph network structure is set to 200. In the graph structure loss function, the hyperparameter t of the matrix M for constructing simulated deep walks is set to 2. The number of training times for each module is set to 50, the learning rate of the optimizer is set to 0.001, the weight decay is set to 0, and the conversion function is set to 1. The default experimental settings are used for other methods.

[0035] The experiment described above uses the Micro-F1 evaluation standard, which is as follows: merge all samples into a large category, calculate the precision and recall, and then calculate Among them: TP is a true positive example, FN is a false negative example, and FP is a false positive example.

[0036] This embodiment selects a total of 7 graph neural network methods as the control group of the present invention, and conducts experiments on the same-matched graphs and different-matched graphs in unbalanced data scenarios, respectively, among which: GCN, GAT, and SGC are classic GNN models for learning nodes of the same-matched graphs. H2GCN and ASGC are two benchmark models for processing different-matched graph structures. RECT is a benchmark model for zero-sample learning under the same-matched graph. By comparing these 7 benchmark models, the performance of the present invention in learning different graph structures in unbalanced data scenarios can be effectively evaluated through specific actual experiments. The experimental results are shown in Tables 2 and 3.

[0037] Table 2 Experimental results with the same diagram under completely unbalanced conditions.

[0038] Table 3 Experimental results of heterogeneous graphs under completely unbalanced conditions.

[0039] The present invention achieved the highest or second highest Micro-F1 value in all five data sets.

[0040] In summary, compared with the prior art, the performance of the present invention in 5 same-matching graph datasets is equal to or even slightly better than all benchmark models, indicating that the present invention can also adaptively handle graph embedding representation learning tasks under the same graph structure.

[0041] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A method for processing heterogeneous graphs based on zero-shot heterogeneous graph embedding based on semantic space mapping, characterized in that: In the offline stage, a RESCN containing an adaptive simple graph convolutional layer is constructed, and the preprocessed dataset is used as a sample for training; In the online stage, the trained RESCN is used to process the heterogeneous graph information received in real time.

2. The method for processing heterogeneous images for zero-sample heterogeneous image embedding according to claim 1 is characterized in that: include: Step 1) Construct an adaptive simple graph convolution layer, set corresponding filters for each different feature, and adapt to the same-matching and different-matching structures by changing the weights of different filters. For each node feature, obtain the feature through ASGC convolution, process the same-matching graph and the different-matching graph at the same time, and adaptively convolve similar features; Step 2) Construct a RESCN including an SVD singular value decomposition layer, an ASGC layer, and a linear layer, wherein: the SVD singular value decomposition layer performs SVD singular value decomposition on the feature matrix in the read feature file to obtain a full rank decomposition of the feature matrix. The ASGC layer aggregates the feature vectors of the graph nodes in an adaptive manner and obtains the aggregated features by calculating the correlation with the features of neighboring nodes. The linear layer projects the aggregated features into the semantic space to obtain a graph embedding for classification tasks; Step 3) Train the constructed RESCN by calculating the semantic description loss function between the predicted semantic description vector of the visible class and the actual class semantic description vector to capture the class semantic information of the graph, and calculate the graph structure representation loss function to capture the information of the graph structure. The node sequence is obtained by random depth walk on the graph structure, and the skip-gram model is used to calculate the maximum likelihood estimate of the node sequence.

3. The method for processing heterogeneous images for zero-sample heterogeneous image embedding according to claim 2, characterized in that: The step 1 specifically includes: 1.1) Set corresponding filters for each different feature, and adapt to the same matching and different matching structures by changing the weights of different filters. Specifically, for each node feature x, the feature x is obtained by ASGC convolution Among them: S is the adjacency matrix after Laplace normalization, K is the hyperparameter that controls the width of the ASGC convolution, and c k is the scaling factor of the node feature x estimated linearly using the k-th step feature aggregation, that is: x = (S 1 x;S 2 x;...;S k x)c,x SGCH =(S 1 x;S 2 x;...;S k x)c; 1.2) Processing the same-matching graph and the different-matching graph at the same time, and adaptively convolving similar features, specifically: when it is a same-matching graph, since the neighbor features of the node are almost the same, the smoothed features will be almost consistent with the SGC; when it is a different-matching graph, for neighbor nodes with different node features, their features will be filtered, thereby effectively avoiding the noise caused by the aggregation features of the different-matching structure; the filter adapts to the feature convolution of the same-matching and different-matching structures by adjusting the weights.

4. The method for processing heterogeneous images for zero-sample heterogeneous image embedding according to claim 2, characterized in that: The step 3 specifically includes: 3.1) Input multiple datasets as training samples into RESCN to obtain the semantic description vector of the predicted visible class; 3.2) By calculating the semantic description loss function between the predicted semantic description vector of the visible class and the actual class semantic description vector in: is the class semantic prediction vector of the i-th node in the semantic space S, is the actual class semantic description vector of the i-th node, which is read by the function get: The feature vector of the node is converted to obtain i is the feature vector of the i-th node; 3.3) Calculate the graph structure representation loss function to capture the information of the graph structure, specifically: Where: M = (A + A * A) / 2, A is the adjacency matrix of the heterogeneous graph, UU' is the embedding obtained by the model, and loss() is the loss function; 3.4) Learn the topological structure of the graph by randomly walking the graph structure to obtain the node sequence, and convert the obtained node sequence w = {v i ...v w }∈Ω is regarded as a word sequence, and the skip-gram model is used to calculate the maximum likelihood estimate of the node sequence Where: r is the range of the depth walk window, the probability Pr(v j |v i ) represents the slave node v i Random walk to node v j The probability of using the softmax function is calculated The deep walk is to perform matrix decomposition on the matrix M, where in: is the adjacency matrix after row normalization, e i is the indicator vector, the i-th item is 1, and the rest are 0; 3.5) After training the model through the semantic description loss function, a representation U with semantic information is generated (1) , retrain the model through the graph structure representation loss function to generate a representation U with graph structure information (2) After that, the two representations are concatenated and normalized to obtain the final node embedding U = Normalize ([U (1) ,U (2) ]), where Normalize(.) is the normalization function.

5. The method for processing heterogeneous images for zero-sample heterogeneous image embedding according to claim 4, characterized in that: The semantic description loss function and the graph structure representation loss function are combined, and the model parameters are optimized by training at both the semantic level and the graph structure level. Specifically, in each training iteration, the semantic description loss and the graph structure representation loss are calculated, and the weighted sum of the two is used as the total loss function. The model parameters are updated through the back-propagation algorithm, so that the model is optimized at two levels at the same time.

Images