Image matching method and system based on optimized adaptive metric learning

By introducing optimized adaptive metric learning network and learning step size adaptive penalty coefficients in deep convolutional networks, the problem of inefficient optimization and limited decision boundary accuracy in triplet loss function is solved, and image feature learning is realized Efficient optimization and improvement of matching accuracy.

CN114463569BActive Publication Date: 2024-11-19WUHAN UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210097534.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-27
Publication Date
2024-11-19
Estimated Expiration
2042-01-27

AI Technical Summary

Technical Problem

The existing triple loss function has the problem of inefficient optimization process and limited accuracy of decision boundaries in image matching, which makes it difficult to achieve optimal inter-class distances at the same time, affecting the efficiency and accuracy of image matching.

Method used

By adding an optimized adaptive metric learning network to the deep convolutional network, using quadratic classification and assigning the penalty weight of triple loss function, introducing the penalty coefficient of learning step size adaptation, optimizing the network parameters, and sample through the division and governance algorithm Classify and refine the spatial distribution relationship.

Benefits of technology

It realizes efficient optimization of image feature learning, improves network matching accuracy and matching efficiency, and improves image matching accuracy without changing the original deep network framework.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114463569B_ABST
    Figure CN114463569B_ABST
Patent Text Reader

Abstract

The present invention discloses an image matching method and system based on optimized adaptive metric learning, and proposes a penalty coefficient that is perceptible to the distance between classes within a class, which is used as an additional optimization coefficient for the distance between two classes, respectively, to construct an optimized adaptive loss function form. At the same time, the divide-and-conquer algorithm is adopted to first perform coarse-grained sample classification on the samples, and then expand the similarity relationship of the samples in the triplet from binary to quaternary, and refine the spatial distribution relationship between different samples, thereby improving the accuracy of sample classification by the entire network, and finally achieving the effect of improving the accuracy of image matching without introducing a new network model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to an image matching method and system, and specifically relates to an image matching method and system based on optimized adaptive metric learning. Background Art

[0002] Image matching is an important research area in the field of artificial intelligence. Many classic pattern recognition problems, including face recognition, image retrieval, and pedestrian re-identification, can be attributed to image matching problems. Most of the current image matching methods rely heavily on deep learning frameworks, and training strategies and loss functions are one of the most important contents in deep learning. The rationality and accuracy of their design will directly affect the deep feature expression of the image, and thus affect the final result of image matching. Metric learning is a commonly used loss function construction method in image matching. The core of the method is to increase the distance between samples between classes and shorten the distance between samples within a class in the sample feature space as the main optimization direction, so as to realize the mapping of samples from the original image space to the feature metric space. Among them, the triple loss function is one of the most widely used loss functions in metric learning. This method can make up for the problem of inaccurate expression of sample features by representation learning to a certain extent, but there are still the following two problems:

[0003] 1) Inefficient optimization process. During back propagation, the gradients of the samples of the triplet loss function for the intra-class distance and the inter-class distance are exactly the same. As a result, when the intra-class distance (or inter-class distance) is close to the optimal state and the inter-class distance (or intra-class distance) is in a suboptimal state, the triplet loss function is difficult to reasonably and efficiently optimize the intra-class and inter-class distances with different learning rates, resulting in limited efficiency of the entire parameter learning process.

[0004] 2) The accuracy of the decision boundary is limited. The optimization boundary of the triplet loss function is approximately parallel to (inter-class distance - intra-class distance). The optimization design of the same penalty coefficient may result in that when (inter-class distance - intra-class distance) takes the optimal value, the inter-class distance and the intra-class distance cannot reach the optimal value at the same time. For example, the intra-class distance = 0.1, the inter-class distance = 0.6 and the intra-class distance = 0.2, the inter-class distance = 0.7. According to the definition of the triplet loss function, the optimization results of the two are the same, but the combination of the two measurement results of intra-class distance = 0.1 and inter-class distance = 0.7 is obviously a more satisfactory optimization result for the intra-class and inter-class distances. Summary of the invention

[0005] To address the above problems, this patent proposes an image matching method and system based on optimized adaptive metric learning.

[0006] The technical solution adopted by the method of the present invention is: an image matching method based on optimized adaptive metric learning, comprising the following steps:

[0007] Step 1: Build an optimized adaptive metric learning network;

[0008] The optimized adaptive metric learning network adds a fully connected layer with an output dimension of K to the last output layer of Dnet to transform the output result into a 1×K vector as the feature vector f of the image. pre On this basis, add a fully connected layer with an output dimension of L, and output a 1×L output result vector V p ;

[0009] Step 2: training the optimized adaptive metric learning network to obtain a trained optimized adaptive metric learning network;

[0010] Among them, the training sample triplet is recorded as Among them, the anchor point sample is recorded as The positive sample is the sample data with the same label as the anchor point, denoted as Negative samples are sample data with different labels from anchor points and are recorded as

[0011] Step 3: Input the image to be matched into the trained optimized adaptive metric learning network and output the matched image.

[0012] The technical solution adopted by the system of the present invention is: an image matching system based on optimized adaptive metric learning, comprising the following modules:

[0013] Module 1, for building an optimized adaptive metric learning network;

[0014] The optimized adaptive metric learning network adds a fully connected layer with an output dimension of K to the last output layer of Dnet to transform the output result into a 1×K vector as the feature vector f of the image. pre On this basis, add a fully connected layer with an output dimension of L, and output a 1×L output result vector V p ;

[0015] Module 2 is used to train the optimized adaptive metric learning network to obtain a trained optimized adaptive metric learning network;

[0016] Among them, the training sample triplet is recorded as Among them, the anchor point sample is recorded as The positive sample is the sample data with the same label as the anchor point, denoted as Negative samples are sample data with different labels from anchor points and are recorded as

[0017] Module 3 is used to input the image to be matched into the trained optimized adaptive metric learning network and output the matched image.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] (1) In the image feature learning stage, the present invention utilizes secondary classification and a method of assigning penalty weights to the triplet loss function to further refine the original sample feature metric space, and realizes efficient optimization of network parameters in the training stage.

[0020] (2) The present invention can be embedded in most deep convolutional network architectures and can optimize the network parameters without changing the original deep network framework, thereby improving the network matching accuracy and being easy to extend to a variety of matching scenarios.

[0021] (3) The present invention can improve the matching efficiency of the original matching framework with less computational complexity by using an optimization step size adaptive strategy without introducing any prior knowledge or additional information. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0024] Metric learning plays an important role in image matching fields including face recognition, image retrieval, and target recognition, mainly in the selection of training strategies and the construction of loss functions. Among them, the method based on the triplet loss function is one of the most commonly used metric learning methods. Its purpose is to achieve the approximation of the spatial distribution of real sample features by optimizing the distance between classes within the sample class. However, the existing triplet loss function mainly faces the following two shortcomings: inefficient optimization process and limited accuracy of decision boundaries.

[0025] Please see Figure 1 The present invention provides an image matching method based on optimized adaptive metric learning, comprising the following steps:

[0026] Step 1: Build an optimized adaptive metric learning network;

[0027] The optimized adaptive metric learning network of this embodiment is to add a fully connected layer with an output dimension of K to the last output layer of Dnet to transform the output result into a 1×K vector as the feature vector f of the image.pre On this basis, add a fully connected layer with an output dimension of L, and output a 1×L output result vector V p ;

[0028] Dnet represents an end-to-end deep convolutional network for feature learning. Dnet can be a custom convolutional network or a classic deep convolutional network such as Resnet, DenseNet, XCeption, etc. The training strategy and loss function based on optimization and adaptation proposed in this patent are applicable to most deep convolutional networks.

[0029] Step 2: Train the optimized adaptive metric learning network to obtain a trained optimized adaptive metric learning network;

[0030] In this embodiment, the specific implementation of step 2 includes the following sub-steps:

[0031] Step 2.1: Construct an image training sample set;

[0032] Known image training sample set P i =(img i ,ID i ), where P i Represents the i-th image sample in the training set, ID i ∈N L is the label vector of the sample, L is the total number of all sample categories in the sample set, and N is the total number of samples; img i Indicates P i The corresponding sample image;

[0033] Step 2.2: Input sample i in the image training sample set into the optimized adaptive metric learning network and output a 1×L result vector

[0034] Step 2.3: Use the Softmax formula to Normalize to vector

[0035]

[0036] Step 2.4: Optimize the network parameters by back propagation using the cross entropy loss function shown in formula (2); after training the network for a preset number of rounds, obtain a trained optimized adaptive metric learning network;

[0037]

[0038] where y il Represents the vector value of the lth dimension in the true label vector of the i-th sample, It is the vector value of the lth dimension of the output label vector of the i-th sample.

[0039] In metric learning based on triplet loss function, the sample group consists of three types of data: anchor samples (anchor) Positive samples: Sample data with the same label as the anchor point Negative samples: Sample data with different labels from anchor points The three types of samples form a triplet The classic triplet loss function (3) optimizes the distance between samples within a class and between classes.

[0040]

[0041] Where f(x) represents the feature vector of sample x, d() represents the Euclidean distance between two feature vectors, m is the decision boundary, [[] + =max{x,0}.

[0042] The distance between classes in formula (3) and intra-class distance The gradient / penalty coefficients are the same, which will cause the learning rates of the intra-class and inter-class distances to remain consistent during back propagation, resulting in low optimization efficiency and inaccurate decision boundaries. Therefore, this patent adds a penalty coefficient α with adaptive learning step size to the intra-class and inter-class distance calculation items. p and α n The hyperparameters OP and ON are introduced to represent and Optimal ideal value, design two penalty coefficients α p and α n , so that α p Positively related to α n Positively related to Expand formula (3) to formula (4):

[0043]

[0044] in β is the scale adjustment parameter.

[0045] In this embodiment, the obtained label set P i ={(img i ,ID i ,M i )}, define two types of negative samples and two types of positive samples. For a given anchor sample Define common negative samples as samples with different IDs and different M from the anchor points, using Indicates that difficult negative samples are defined as samples with different IDs and the same M, using Indicates; define a normal positive sample as a sample with the same ID and the same M as the anchor point, and use It means that the difficult positive samples are defined as samples with the same ID and different M as the anchor point, and According to α p and α n The definition of , assigns different optimal ideal values ​​OP and ON to different categories of positive and negative samples. According to the visual similarity between image samples, the optimal distance ON between difficult negative samples and anchor points h Should be slightly smaller than the optimal distance between common negative samples and anchor points ON n ; The optimal distance OP between the difficult positive sample and the anchor point h Should be slightly greater than the optimal distance OP between the normal positive sample and the anchor point n , so set ON h =δ N ON n , OP n =δ P OP p , δ N ,δ P ∈(0,1).

[0046] In this embodiment, bsize·L samples are randomly selected as anchor points in the training set. The selected samples satisfy all (L) label categories, and the number of samples selected for each label is bsize, where bsize is a preset hyperparameter, which is set to 1 by default in the experiment, and L is the total number of label categories in the training sample. Construct bsize·L triplets, and use the triple loss function L proposed in this patent T Further parameter optimization is performed on Dnet, where Dnet is the optimized adaptive metric learning network A trained in step 2.

[0047] Retraining the optimized adaptive metric learning network A to obtain a trained optimized adaptive metric learning network;

[0048] The specific implementation includes the following sub-steps:

[0049] Step 2.5: Input sample i in the image training sample set into the optimized adaptive metric learning network A, and output the feature vector set of the image to be matched as Map samples from image space to initial feature space;

[0050] Step 2.6: Use K-Means clustering method to Clustering is performed using the Euclidean distance metric The distance between any two points, the cluster category is preset to W; randomly initialize W sample points as the class center of W clusters, calculate the distance from several data points to the center point, and the category of the data point is defined as A i If and only if the distance between the data point and the center point of the category is smaller than the distance between the data point and other center points; use the multidimensional space category center calculation formula (vector average) to calculate the center point of the new category, and repeat until the center of each category no longer changes, and get the clustering result;

[0051] Step 2.7: According to the clustering results of step 2.6, re-label each image to be matched so that the original label data P i =(img i ,ID i ) extends to P i =(img i ,ID i ,M i ), where M i is img i The cluster category it belongs to can be understood as img i coarse-grained classification categories;

[0052] but middle, img ia Representation sample The corresponding sample image, ID ia yes The real ID tag, M ia It is obtained by classifying the samples by network A in step 2.7 The corresponding tag. img ip Representation sample The corresponding sample image, ID ip yes The real ID tag, M ip It is obtained by classifying the samples by network A in step 2.7 The corresponding tag. img in Representation sample The corresponding sample image, ID in yes The real ID tag, M in It is obtained by classifying the samples by network A in step 2.7 The corresponding label.

[0053] Step 2.8: For a given anchor point sample Define common negative samples as samples with different IDs and different M from the anchor points, using Indicates that difficult negative samples are defined as samples with different IDs and the same M, using Indicates; define a normal positive sample as a sample with the same ID and the same M as the anchor point, and use It means that the difficult positive samples are defined as samples with the same ID and different M as the anchor point, and express;

[0054] According to the visual similarity between image samples, the optimal distance between difficult negative samples and anchor points is ON h Should be slightly smaller than the optimal distance between common negative samples and anchor points ON n ; The optimal distance OP between the difficult positive sample and the anchor point h Should be slightly greater than the optimal distance OP between the normal positive sample and the anchor point n , ON h =δ N ON n , OP n =δ P OP p , δ N ,δ P ∈(0,1);

[0055] Use difficult samples, common samples and loss function L T Retraining the optimized adaptive metric learning network A to obtain a trained optimized adaptive metric learning network;

[0056]

[0057]

[0058]

[0059] Where f(x) represents the feature vector of sample x, d() represents the Euclidean distance between two feature vectors, m is the decision boundary, [x] + =max{x,0};α p and α n To learn the penalty coefficient for step size adaptation, the hyperparameters OP and ON represent and The optimal ideal value; β is the scale adjustment parameter.

[0060] Step 3: Input the image to be matched into the trained optimized adaptive metric learning network and output the matching image;

[0061] This patent proposes a penalty coefficient that is perceptible to the distance between classes within a class, which is used as an additional optimization coefficient for the two types of distances to construct an optimized adaptive loss function. At the same time, the divide-and-conquer algorithm is adopted to first perform coarse-grained sample classification on the samples, expand the similarity relationship of the samples in the triplet from binary to quaternary, refine the spatial distribution relationship between different samples, and then improve the accuracy of sample classification of the entire network, and finally achieve the effect of improving the accuracy of image matching without introducing a new network model.

[0062] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. An image matching method based on optimized adaptive metric learning, It is characterized in that The following steps are involved: Step 1: Build an optimized adaptive metric learning network; The optimized adaptive metric learning network adds a fully connected layer with an output dimension of K to the last output layer of Dnet to transform the output result into a 1×K vector as the feature vector f of the image. pre On this basis, add a fully connected layer with an output dimension of L, and output a 1×L output result vector V p ; Step 2: training the optimized adaptive metric learning network to obtain a trained optimized adaptive metric learning network; Among them, the training sample triplet is recorded as Among them, the anchor point sample is recorded as The positive sample is the sample data with the same label as the anchor point, denoted as Negative samples are sample data with different labels from anchor points and are recorded as The specific implementation of step 2 includes the following sub-steps: Step 2.1: Construct an image training sample set; Known image training sample set P i =(img i ,ID i ), where P i Represents the i-th image sample in the training set, ID i ∈N L is the label vector of the sample, L is the total number of all sample categories in the sample set, and N is the total number of samples; img i Indicates P i The corresponding sample image; Step 2.2: Input sample i in the image training sample set into the optimized adaptive metric learning network and output a 1×L result vector Step 2.3: Use the Softmax formula to Normalize to vector Step 2.4: Optimize the network parameters by back propagation using the cross entropy loss function shown in formula (2); after the network is trained for a preset number of rounds, a trained optimized adaptive metric learning network A is obtained; where y il Represents the vector value of the lth dimension in the true label vector of the i-th sample, is the vector value of the lth dimension of the output label vector of the i-th sample; Retraining the optimized adaptive metric learning network A to obtain a trained optimized adaptive metric learning network; The specific implementation includes the following sub-steps: Step 2.5: Input sample i in the image training sample set into the optimized adaptive metric learning network A, and output the feature vector set of the image to be matched as Map samples from image space to initial feature space; Step 2.6: Use K-Means clustering method to Clustering is performed using the Euclidean distance metric The distance between any two points, the cluster category is preset to W; randomly initialize W sample points as the class center of W clusters, calculate the distance from several data points to the center point, and the category of the data point is defined as A i If and only if the distance between the data point and the center point of the category is smaller than the distance between the data point and other center points; use the multidimensional space category center calculation formula to calculate the center point of the new category, and repeat the process until the center of each category no longer changes, and get the clustering result; Step 2.7: According to the clustering results of step 2.6, re-label each image to be matched so that the original label data P i =(img i ,ID i ) extends to P i =(img i ,ID i ,M i ), where M i is img i The cluster category it belongs to is img i coarse-grained classification categories; Step 2.8: For a given anchor point sample Define common negative samples as samples with different IDs and different M from the anchor points, using Indicates that difficult negative samples are defined as samples with different IDs and the same M, using Indicates; define a normal positive sample as a sample with the same ID and the same M as the anchor point, and use It means that the difficult positive samples are defined as samples with the same ID and different M as the anchor point, and express; According to the visual similarity between image samples, the optimal distance between difficult negative samples and anchor points is ON h Should be slightly smaller than the optimal distance between common negative samples and anchor points ON n ; The optimal distance OP between the difficult positive sample and the anchor point h Should be slightly greater than the optimal distance OP between the normal positive sample and the anchor point n , ON h =δ N ON n , OP n =δ P OP p , δ N ,δ P ∈(0,1); Use difficult samples, common samples and loss function L T Retraining the optimized adaptive metric learning network A to obtain a trained optimized adaptive metric learning network; Where f(x) represents the feature vector of sample x, d() represents the Euclidean distance between two feature vectors, m is the decision boundary, [x] + =max{x,0};α p and α n To learn the penalty coefficient for step size adaptation, the hyperparameters OP and ON represent and The optimal ideal value; β is the scale adjustment parameter; Step 3: Input the image to be matched into the trained optimized adaptive metric learning network and output the matched image.

2. An image matching system based on optimized adaptive metric learning, used to implement the method described in claim 1; It is characterized in that Includes the following modules: Module 1, for building an optimized adaptive metric learning network; The optimized adaptive metric learning network adds a fully connected layer with an output dimension of K to the last output layer of Dnet to transform the output result into a 1×K vector as the feature vector f of the image. pre On this basis, add a fully connected layer with an output dimension of L, and output a 1×L output result vector V p ; Module 2 is used to train the optimized adaptive metric learning network to obtain a trained optimized adaptive metric learning network; Among them, the training sample triplet is recorded as Among them, the anchor point sample is recorded as The positive sample is the sample data with the same label as the anchor point, denoted as Negative samples are sample data with different labels from anchor points and are recorded as Module 3 is used to input the image to be matched into the trained optimized adaptive metric learning network and output the matched image.

Citation Information

Patent Citations

  • Fine-grained image retrieval method based on comparison similarity loss function

    CN110659378A

  • Method and electronic device for retrieving an image and computer readable storage medium

    US20200242422A1

  • Image classification method based on reliable weighted optimal transport (RWOT)

    US20210390355A1

  • Image recognition system and image recognition method

    WO2018119684A1