Unsupervised graph matching method based on two-stage self-paced learning
By using a two-level self-step learning method in the graph matching model, the node characteristics and sample sets are updated, and the problems of low confidence nodes and random sample subsets in the existing technology are solved, which significantly improves the graph matching accuracy.
Patent Information
- Application Number
- CN202411200312.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-19
AI Technical Summary
In the prior art, the graph matching model uses node characteristics that do not distinguish the confidence of nodes during the training process, resulting in low confidence nodes affecting the matching accuracy. The random selection of the training sample subset ignores the confidence difference of the sample itself, limiting the graph. Improved matching accuracy.
Using the unsupervised graph matching method based on two-level self-step learning, the network updates the node features through self-step learning sub-graph updates using the indication matrix with a high confidence value, and updates the training samples based on confidence through the self-step learning feedback network of batch samples. Set, avoiding the defects of low confidence nodes and random sample subsets.
It effectively improves the graph matching accuracy, avoids the accuracy limitations caused by low confidence node characteristics and random sample subset selection, and improves the accuracy of matching results.
Smart Images

Figure CN118982693A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of graph data mining and relates to a graph matching method, specifically to an unsupervised graph matching method based on two-level self-paced learning, which can be used in the fields of computer vision, social network analysis, molecular property prediction, etc. Background Art
[0002] Graph matching is a method of finding the node correspondence between two or more graph data using the structural information of the graph data structure. Due to the large scale of graph data in real life, using machine learning to obtain graph data information for matching has become a hot topic. Machine learning has powerful feature extraction and learning capabilities, and can better extract node representations and high-level representations of graph data.
[0003] The graph matching method based on machine learning mainly extracts features from images and constructs graph data based on neural networks, and then solves the node correspondence of graph data through machine learning models, and finally realizes graph matching. In order to improve the accuracy of graph matching, Liu Chang published a paper titled "Self-supervised Learning of Visual Graph Matching" at the conference "European Conference on Computer Vision" in October 2022, which disclosed an end-to-end self-supervised contrast graph matching method. This method performs data augmentation on the original image to generate its corresponding enhanced image, and extracts node features to construct the original graph and enhanced graph structure; then uses the node update network to update the original graph and enhanced graph node representation; uses the node matching network to calculate the node matching labels for the updated original graph node features and enhanced graph node features; calculates the loss value by comparing the learning loss and the interpolation matching loss function, optimizes the model, and obtains the trained SCGM graph matching model; inputs the test sample into the trained graph matching model to obtain the predicted graph matching result. This method has a high matching accuracy. However, the shortcoming of this method is that in the process of training the SCGM graph matching model, the node features that do not distinguish between the high and low node confidences are used to update the model, resulting in the existence of low-confidence nodes in the updating process, which affects the improvement of matching accuracy; and the training sample subset is randomly selected, ignoring the confidence differences of the samples themselves, which further limits the improvement of graph matching accuracy. Summary of the invention
[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose an unsupervised graph matching method based on two-level self-paced learning to solve the technical problem of low matching accuracy in the prior art.
[0005] To achieve the above object, the technical solution adopted by the present invention includes the following steps:
[0006] (1) Obtain training sample set and test sample set:
[0007] Obtain I natural images including M object categories, and preprocess each natural image. Then, each two natural images of each category in the preprocessed I natural images containing N nodes form a sample. Then, more than half of the samples are evenly divided into S groups, and the multiple samples included in each group are formed into training sample subsets to obtain a training sample set P consisting of S training sample subsets. The remaining samples form a test sample set.
[0008] Among them, M≥2, I≥500, N≥20, S≥100;
[0009] (2) Construct an unsupervised graph matching network model O based on two-level self-paced learning:
[0010] Constructing an unsupervised graph matching network model O including a cascaded initial graph construction network, an associated graph construction network, a self-paced learning subgraph update network, a node matching network and a batch sample self-paced learning feedback network, wherein the output end of the batch sample self-paced learning feedback network is also connected to the input end of the feature extraction network; wherein the self-paced learning subgraph update network includes a cascaded self-paced learning subgraph construction module and a subgraph node representation update module, which is used to update the node features of the fused associated subgraph and the node features of the assigned associated subgraph through an indicator matrix with a higher confidence value; the batch sample self-paced learning feedback network includes a cascaded loss storage module and a batch sample reordering module, which is used to update the training sample set through confidence;
[0011] (3) Iterative training of the unsupervised graph matching network model:
[0012] The unsupervised graph matching network model O is iteratively trained through the training sample set P to obtain the trained unsupervised graph matching network model O * ;
[0013] (4) Obtain unsupervised graph matching results:
[0014] The test sample set is used as the trained unsupervised graph matching network model O * The input is forward propagated, and the output of the node matching network is used as the matching result of the test sample set.
[0015] Compared with the prior art, the present invention has the following advantages:
[0016] In the process of training the unsupervised graph matching network model, the present invention uses a self-paced learning subgraph update network to update the node features of the fused associated subgraph and the node features of the allocated associated subgraph through an indicator matrix with a higher confidence value, and a batch sample self-paced learning feedback network to update the training sample set through the confidence value, and uses the updated result as the input for the next iteration, thereby avoiding the defects of the prior art due to the existence of node features with low confidence and the random selection of training sample subsets, and effectively improving the matching accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a flow chart for implementing the present invention.
[0018] Figure 2 It is a structural schematic diagram of the matching network model of the present invention. DETAILED DESCRIPTION
[0019] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0020] Reference Figure 1 , the present invention comprises the following steps:
[0021] Step 1) Obtain training sample set and test sample set:
[0022] For I natural images including M object categories obtained from the WILLOW OBJECT CLASS dataset, each natural image is cropped with the object as the center, and N pixels in each cropped natural image are randomly selected and marked as nodes, so as to obtain I natural images each containing N nodes after preprocessing. Then, every two natural images of each category in the I natural images each containing N nodes after preprocessing are used as a sample, and then more than half of the samples are evenly divided into S groups, and the multiple samples included in each group are used as training sample subsets to obtain a training sample set P composed of S training sample subsets, and the remaining samples are used as a test sample set. In this embodiment, M=5, which are duck, car, face, motorcycle, and beer bottle, respectively, I=500, S=100, and N=20;
[0023] Step 2) Construct an unsupervised graph matching network model O based on two-level self-paced learning, whose structure is as follows: Figure 2 As shown:
[0024] Constructing an unsupervised graph matching network model O including a cascaded initial graph construction network, an associated graph construction network, a self-paced learning subgraph update network, a node matching network and a batch sample self-paced learning feedback network, wherein the output end of the batch sample self-paced learning feedback network is also connected to the input end of the feature extraction network; wherein the self-paced learning subgraph update network includes a cascaded self-paced learning subgraph construction module and a subgraph node representation update module, which is used to update the node features of the fused associated subgraph and the node features of the assigned associated subgraph through an indicator matrix with a higher confidence value; the batch sample self-paced learning feedback network includes a cascaded loss storage module and a batch sample reordering module, which is used to update the training sample set through confidence;
[0025] Initial graph construction network, including parallel node feature extraction module and adjacency matrix construction module;
[0026] An association graph construction network, including a parallel fusion association graph construction module and a distribution association graph construction module;
[0027] The self-paced learning subgraph construction module in the self-paced learning subgraph update network includes a parallel fusion-related subgraph construction module and an allocation-related subgraph construction module; the subgraph node representation update module includes a parallel fusion-related subgraph node representation update module and an allocation-related subgraph node representation update module;
[0028] The node matching network includes a parallel fusion associated subgraph node matching module and a distribution associated subgraph node matching module.
[0029] Step 3) Iteratively train the unsupervised graph matching network model:
[0030] The unsupervised graph matching network model O is iteratively trained through the training sample set P to obtain the trained unsupervised graph matching network model O * ;
[0031] (3a) The number of initial iterations is t, the maximum number of iterations is T, T ≥ 100, and the unsupervised graph matching model O corresponding to the s-th training sample subset s The learnable weights and bias parameters are w s 、b s , and let s = 1, t = 1; in this embodiment, let T = 200;
[0032] (3b) The initial graph construction network extracts the features υ1 and υ2 of the two image nodes of each training sample in each training sample subset, and constructs the adjacency matrix A1 and adjacency matrix A2 of the two image nodes;
[0033] The node feature extraction module in the initial graph construction network extracts node features from the two image nodes of each training sample in the sth training sample subset to obtain the node features υ1 of the first image and the node features υ2 of the second image. At the same time, the adjacency matrix construction module constructs edges for the nodes of the two images to obtain the adjacency matrix A1 and the adjacency matrix A2.
[0034] The fusion association graph construction module in the association graph construction module calculates the node features of the fusion graph through the node features υ1 and υ2 And by fusing the edge similarity K e and the fusion node similarity K p Compute the adjacency matrix of the fused graph At the same time, the allocation association graph construction module calculates the node features of the allocation graph through node features υ1 and υ2 And calculate the adjacency matrix of the distribution graph through A1 and A2 in:
[0035]
[0036] Where [:] represents the concatenation operation, diag(.) represents the diagonalization operation, represents the Kronecker product operation, G1 and H1 represent the edge index matrix of the adjacency matrix A1, G2 and H2 represent the edge index matrix of the adjacency matrix A2, Respectively represent the transposed results of H1 and H2, They respectively represent the transposed results of B2.
[0037] The self-paced learning subgraph update network is right To update, the implementation steps are:
[0038] The fusion correlation subgraph construction module in the self-paced learning subgraph construction module fuses the node features of the correlation graph Calculate the node confidence matrix M1 of the fusion association graph, and use M1 to calculate the node confidence matrix containing the highest front Indicator matrix of confidence values Then pass and as well as and Calculate node features of fused associated subgraphs and the adjacency matrix At the same time, the associated subgraph construction module is allocated by assigning node features of the associated graph Calculate the node confidence matrix M2 of the fused association graph, and use M2 to calculate the node confidence matrix containing the highest front Indicator matrix of confidence values Then pass and as well as and Compute node features of the distribution-associated subgraph and the adjacency matrix Subgraph nodes represent the fusion association in the update module. Subgraph nodes represent the update module through Node Features Update to obtain the updated fusion-related subgraph node features At the same time, the associated subgraph nodes are assigned to indicate that the update module is updated through Node Features Update to obtain the updated distribution-related subgraph node features in:
[0039]
[0040] Among them, softmax() represents the normalization operation, f c1 () indicates a full connection operation, Indicates the selection of the top confidence matrix The largest value, Respectively The transposed result of diag() -1 It means to find the inverse and perform diagonalization operation. The dimension is The identity matrix, f c2 () indicates a full connection operation.
[0041] The node matching network updates the node features of the fused associated subgraphs. Compute matching labels for fused associated subgraphs At the same time, the node features of the associated subgraph are updated through the allocation Compute matching labels for assigned associated subgraphs
[0042] Fusion of matching labels of related subgraphs and assign matching labels to associated subgraphs The calculation formulas are:
[0043]
[0044] where f c3 () and f c4 () represents the fully connected operation, and σ represents the relu activation function.
[0045] In the process of training the unsupervised graph matching network model, the self-learning subgraph update network updates the node features of the fused associated subgraph and the node features of the allocated associated subgraph through an indicator matrix with a higher confidence value, thereby avoiding the defects of the prior art due to the existence of node features with low confidence values and effectively improving the matching accuracy.
[0046] (3c) Using the Frobenius norm function and matching the labels and Compute the unsupervised graph matching model O s The loss value L s , and then use the chain rule to calculate L s For the weight parameter w s and the bias parameter b s Update for a total of S rounds to obtain the unsupervised graph matching model O for the tth iteration t ;
[0047] The unsupervised graph matching model O s The loss value L s , and the weight parameter w s and the bias parameter b s , and its calculation and update formulas are:
[0048]
[0049] where ||.|| F represents the Frobenius norm, which represents the and The absolute value of the difference of each item in is squared and summed up, and then the root is taken, w s ', b s ' indicates O s The weight and bias parameter w s 、b s The update result of , α represents the learning rate.
[0050] (3d) Batch sample self-paced learning feedback network by matching labels and Calculate the confidence r of the sth batch of training sample subsets s , and sort the S confidences in descending order to form a confidence vector R, and then use R to calculate the corresponding S training sample subsets P * Sort and obtain the updated training sample set P;
[0051] In the process of training the unsupervised graph matching network model, the present invention uses a self-paced learning subgraph update network to update the node features of the fused associated subgraph and the node features of the allocated associated subgraph through an indicator matrix with a higher confidence value, and a batch sample self-paced learning feedback network to update the training sample set through the confidence value, and uses the updated result as the input for the next iteration, thereby avoiding the defects of the prior art due to the existence of node features with low confidence and the random selection of training sample subsets, and effectively improving the matching accuracy.
[0052] (3e) Determine whether t>T holds. If so, obtain the trained unsupervised graph matching network model O * , otherwise, let t = t + 1, P * =P, and execute step (3b).
[0053] Step 4) Get the unsupervised graph matching results:
[0054] The test sample set is used as the trained unsupervised graph matching network model O * The input is forward propagated, and the output of the node matching network is used as the matching result of the test sample set.
Claims
1. An unsupervised graph matching method based on two-level self-paced learning, characterized in that: The following steps are involved: (1) Obtain training sample set and test sample set: Obtain I natural images including M object categories, and preprocess each natural image. Then, each two natural images of each category in the preprocessed I natural images containing N nodes form a sample. Then, more than half of the samples are evenly divided into S groups, and the multiple samples included in each group are formed into training sample subsets to obtain a training sample set P consisting of S training sample subsets. The remaining samples form a test sample set, where M≥2, I≥500, N ≥ 20, S ≥ 100; (2) Construct an unsupervised graph matching network model O based on two-level self-paced learning: Constructing an unsupervised graph matching network model O including a cascaded initial graph construction network, an associated graph construction network, a self-paced learning subgraph update network, a node matching network and a batch sample self-paced learning feedback network, wherein the output end of the batch sample self-paced learning feedback network is also connected to the input end of the feature extraction network; wherein the self-paced learning subgraph update network includes a cascaded self-paced learning subgraph construction module and a subgraph node representation update module, which is used to update the node features of the fused associated subgraph and the node features of the assigned associated subgraph through an indicator matrix with a higher confidence value; the batch sample self-paced learning feedback network includes a cascaded loss storage module and a batch sample reordering module, which is used to update the training sample set through confidence; (3) Iterative training of the unsupervised graph matching network model: Through the training sample set P * Iteratively train the unsupervised graph matching network model O to obtain the trained unsupervised graph matching network model O * ; (4) Obtain unsupervised graph matching results: The test sample set is used as the trained unsupervised graph matching network model O * The input is forward propagated, and the output of the node matching network is used as the matching result of the test sample set.
2. The method according to claim 1, characterized in that: The preprocessing of each natural image in step (1) is implemented as follows: Each natural image is cropped with the object as the center, and N pixels in each cropped natural image are randomly selected and marked as nodes, so as to obtain I natural images each containing N nodes after preprocessing.
3. The method according to claim 1, characterized in that The unsupervised graph matching network model O described in step (2), wherein: Initial graph construction network, including parallel node feature extraction module and adjacency matrix construction module; The association graph construction network includes a parallel fusion association graph construction module and a distribution association graph construction module. The self-paced learning subgraph construction module in the self-paced learning subgraph update network includes a parallel fusion-related subgraph construction module and an allocation-related subgraph construction module; the subgraph node representation update module includes a parallel fusion-related subgraph node representation update module and an allocation-related subgraph node representation update module; The node matching network includes a parallel fusion associated subgraph node matching module and a distribution associated subgraph node matching module.
4. The method according to claim 3, characterized in that: The unsupervised graph matching network model described in step (3) is iteratively trained, and the implementation steps are as follows: (3a) The number of initial iterations is t, the maximum number of iterations is T, T ≥ 100, and the unsupervised graph matching model O corresponding to the s-th training sample subset s The learnable weights and bias parameters are w s , b s , and let s = 1, t = 1; (3b) The initial graph construction network extracts the features υ1 and υ2 of the two image nodes of each training sample in each training sample subset, and constructs the adjacency matrix A1 and adjacency matrix A2 of the two image nodes; the association graph construction network calculates the node features of the fused association graph through υ1 and A1 and the adjacency matrix The node features of the distribution association graph are calculated by υ2 and A2 at the same time and the adjacency matrix The self-paced learning subgraph update network is right Update; the node matching network is updated through the node features of the fusion-associated subgraph Compute matching labels for fused associated subgraphs At the same time, the node features of the associated subgraph are updated through the allocation Compute matching labels for assigned associated subgraphs (3c) Using the Frobenius norm function and matching the labels and Compute the unsupervised graph matching model O s The loss value L s , and then use the chain rule to calculate L s For the weight parameter w s and the bias parameter b s Update for a total of S rounds to obtain the unsupervised graph matching model O for the tth iteration t ; (3d) Batch sample self-paced learning feedback network by matching labels and Calculate the confidence r of the sth training sample subset s , and sort the S confidences into confidence vectors in descending order, and then sort the S training sample subsets P corresponding to the confidence vectors to obtain the updated training sample set P * ; (3e) Determine whether t>T holds. If so, obtain the trained unsupervised graph matching network model O * Otherwise, let t = t + 1, P = P * , and execute step (3b).
5. The method according to claim 4, characterized in that The features υ1 and υ2 of the two image nodes described in step (3b), as well as the adjacency matrix A1 and the adjacency matrix A2, are obtained by: The node feature extraction module in the initial graph construction network extracts node features from the two image nodes of each training sample in the sth training sample subset to obtain the node features υ1 of the first image and the node features υ2 of the second image. At the same time, the adjacency matrix construction module constructs edges for the nodes of the two images to obtain the adjacency matrix A1 and the adjacency matrix A2.
6. The method according to claim 4, characterized in that Node features of the fused association graph described in step (3b) and the adjacency matrix And the node characteristics of the distribution association graph and the adjacency matrix The calculation method is: The fusion association graph construction module in the association graph construction module calculates the node features of the fusion graph through the node features υ1 and υ2 And by fusing the edge similarity K e and the fusion node similarity K p Compute the adjacency matrix of the fused graph At the same time, the allocation association graph construction module calculates the node features of the allocation graph through node features υ1 and υ2 And calculate the adjacency matrix of the distribution graph through A1 and A2 in: Where [:] represents the concatenation operation, diag(.) represents the diagonalization operation, represents the Kronecker product operation, V1 and H1 represent the edge index matrices of the adjacency matrix A1, and V2 and H2 represent the edge index matrices of the adjacency matrix A2.
7. The method according to claim 4, characterized in that The self-learning subgraph update network described in step (3b) is and as well as and To update, the implementation method is: The fusion correlation subgraph construction module in the self-paced learning subgraph construction module fuses the node features of the correlation graph Calculate the node confidence matrix M1 of the fusion association graph, and use M1 to calculate the node confidence matrix containing the highest front Indicator matrix of confidence values Then pass and as well as and Calculate node features of fused associated subgraphs and the adjacency matrix At the same time, the associated subgraph construction module is allocated by assigning node features of the associated graph Calculate the node confidence matrix M2 of the fused association graph, and use M2 to calculate the node confidence matrix containing the highest front Indicator matrix of confidence values Then pass and as well as and Compute node features of the distribution-associated subgraph and the adjacency matrix Subgraph nodes represent the fusion association in the update module. Subgraph nodes represent the update module through Node Features Update to obtain the updated fusion-related subgraph node features At the same time, the associated subgraph nodes are assigned to indicate that the update module is updated through Node Features Update to obtain the updated distribution-related subgraph node features in: Among them, softmax() represents the normalization operation, f c1 () indicates a full connection operation, Indicates the selection of the top confidence matrix The largest value, Respectively The transposed result of diag() -1 It means to find the inverse and perform diagonalization operation. The dimension is The identity matrix, f c2 () indicates a full connection operation.
8. The method according to claim 4, characterized in that The matching labels of the fused correlation subgraph described in step (3b) and assign matching labels to associated subgraphs The calculation formulas are: where f c3 () and f c4 () represents the fully connected operation, and σ represents the relu activation function.
9. The method according to claim 4, characterized in that The unsupervised graph matching model O described in step (3c) s The loss value L s , and the weight parameter w s and the bias parameter b s , and its calculation and update formulas are: where ||.|| F represents the operation of taking the Frobenius norm, w s ', b s ' indicates O s The weight and bias parameter w s , b s The update result is , α represents the learning rate.
