Image Description and Matching Method Based on Sample Mixing and Improved Trivariate Loss Function

Abstract

This invention discloses an image feature point description method based on sample mixing and an improved ternary loss function, comprising the following steps: S1, keypoint detection is performed on a given image; S2, image patches of the same size are extracted, centered on each keypoint; S3, the extracted image patches are input into a pre-trained convolutional neural network for image feature description, and the feature descriptor vectors of all keypoints are output. This invention considers the diversity of sample distances and utilizes the classification of sample feature vectors to further increase the accuracy of sample feature description, thereby effectively improving the accuracy of image matching.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and more particularly relates to an image feature point matching method and system based on sample mixing and an improved ternary loss function. BACKGROUND

[0002] Image feature extraction belongs to the category of image analysis, is a deep understanding of image information, is an advanced stage of digital image processing, and is also the beginning of image recognition. Image feature extraction technology is crucial in motion target tracking, object recognition, image registration, panoramic image stitching, three-dimensional reconstruction and the like, and the underlying of these applications is based on accurate extraction and precise matching of feature points, and the quality of feature extraction will directly determine the accuracy of practical application in the above-mentioned directions. Therefore, fast and accurate feature point extraction is a basic requirement for realizing upper-layer application.

[0003] Early image feature extraction methods are mostly based on manually designed features, and the manual method focuses on low-level processes such as gradient filters, and local descriptors must be manually designed according to the characteristics of the data. In recent years, with the rapid development of computer hardware such as GPU and the wide application of deep learning in image processing, image feature extraction algorithms based on convolutional neural networks have also been proposed. The key of these learning-based feature extraction algorithms is to design a suitable loss function for the training process, but the existing loss functions still have many shortcomings. Taking the ternary loss function as an example, the distance relationship between the positive sample and the remaining non-matching samples is not fully utilized when selecting negative samples; in addition, sample information may be missing when extracting features from samples, and the existing method does not classify the extracted features, lacking constraints on the classification results. SUMMARY

[0004] The main purpose of the present application is to provide an image feature point description and matching method based on sample mixing and improved ternary loss function. The image feature description vector extracted by the method can make the points with matching each other more compact and the points with no matching each other more distant, thereby improving the image key point matching precision.

[0005] The technical scheme adopted by the present application is as follows:

[0006] Provided is an image feature point description method based on sample mixing and improved ternary loss function, comprising the following steps:

[0007] S1, detecting key points for a given image;

[0008] S2, extracting image blocks of the same size respectively with each key point as the center;

[0009] S3, inputting the extracted plurality of image blocks into the trained convolutional neural network for image feature description, and outputting a feature descriptor vector of all key points;

[0010] The training process of the convolutional neural network is as follows:

[0011] Step 1: constructing image samples containing several categories, generating a triple set (A, P, N), the anchor sample set A includes n anchor samples, the positive sample set P includes n positive samples, and the negative sample set N includes n negative samples; each anchor sample has a positive sample forming a matching pair;

[0012] Step 2: inputting the triple into the pre-constructed convolutional neural network for image feature description, extracting an anchor sample feature vector set and a positive sample feature vector set , and calculating a distance matrix between the positive samples and the anchor samples;

[0013] Step 3: selecting two negative sample sets closest and second closest to each anchor sample within a preset threshold range from the distance matrix , mixing them with the positive sample set according to a certain proportion to form a final synthetic negative sample set, replacing the negative sample set in the triple set, and calculating a triple loss function ;

[0014] Step 4: concatenating the elements in the anchor sample feature vector set output by the convolutional neural network with the elements in the positive sample feature vector set and the elements in the negative sample feature vector set in sequence to obtain two sets and , classifying them, and calculating a cross-entropy loss function according to the classification probability;

[0015] Step 5: mixing the triple loss function and the cross-entropy loss function according to a certain proportion as a total loss function, and then constraining the training process of the convolutional neural network according to the total loss function, and obtaining the expected convolutional neural network after multiple training.

[0016] According to the above technical solution, step 4 further includes:

[0017] labeling the feature vectors obtained after concatenation, wherein the feature vectors concatenated from the anchor sample and the positive sample are defined as positive classes, and are labeled as label=1; the feature vectors concatenated from the anchor sample and the negative sample are defined as negative classes, and are labeled as label=0;

[0018] two sets of labeled and two fully connected layers in succession to achieve further fusion of features;

[0019] The output of the fully connected layer is converted by a softmax activation function to obtain the probability that sample i is predicted as a positive class, denoted as .

[0020] The cross-entropy loss function is used to quantify the distance between the output probability of the network and the expected value, and the calculation formula of the cross-entropy loss function is as follows:

[0021]

[0022] denotes the actual label of sample i, and the positive class is 1 and the negative class is 0, denotes the probability that sample i is predicted as a positive class.

[0023] According to the above technical solution, step Step3 is specifically:

[0024] From the distance matrix , select two negative sample sets that are closest and second closest to each anchor sample within a predetermined threshold range, mix the two negative sample sets according to a certain ratio for the first time to form an intermediate negative sample set, and then mix the positive sample set and the intermediate negative sample set according to a ratio for the second time to generate a final synthetic negative sample set. Replace the negative sample set N in the original triple set (A, P, N) with the final synthetic negative sample set, and calculate the triple loss function .

[0025] According to the above technical solution, when the two negative sample sets in step Step3 are mixed according to a certain ratio for the first time, the mixing process is specifically , wherein is the intermediate negative sample, is the closest negative sample, is the second closest negative sample, wherein .

[0026] According to the above technical solution, when the positive sample set and the intermediate negative sample set are mixed according to a ratio for the second time in step Step3, the mixing process is specifically , wherein is the final synthetic negative sample, , wherein is the distance between each matching pair, .

[0027] According to the technical scheme, the ternary loss function and the cross-entropy loss function are mixed according to a certain proportion as a total loss function in step Step5 The specific mixing formula is as follows:

[0028] , .

[0029] The application further provides an image feature point matching method, comprising the following steps:

[0030] The feature descriptor vectors of two images to be matched are obtained by the method according to the technical scheme.

[0031] The distance between the key points in the two images is calculated according to the feature descriptor vectors.

[0032] The key point matching pairs of the two images are found by matching according to the distance between the key points.

[0033] According to the technical scheme, the calculated distance between the key points is the Euclidean distance.

[0034] The application further provides an image feature point description system based on sample mixing and improved ternary loss function, comprising:

[0035] A key point detection module is configured to detect key points in a given image.

[0036] An image block extraction module is configured to extract image blocks of the same size centered on each key point.

[0037] A feature description module is configured to input the extracted image blocks into a trained convolutional neural network for image feature description, and output feature descriptor vectors of all key points.

[0038] The application further provides an image feature point matching system, comprising:

[0039] A feature description module is configured to obtain feature descriptor vectors of two images to be matched by the method according to the technical scheme.

[0040] A distance calculation module is configured to calculate the distance between the key points in the two images according to the feature descriptor vectors.

[0041] A matching module is configured to find key point matching pairs of the two images by matching according to the distance between the key points.

[0042] The application further provides a computer storage medium, which stores a computer program executable by a processor, and the computer program executes the image feature point description method based on sample mixing and improved ternary loss function.

[0043] The application has the advantages that: considering that the existing method does not necessarily contain all kinds when selecting samples, the distribution range of samples needs to be expanded to overcome the limitation caused by sample sparsity, the negative sample mixing generation strategy is introduced into the existing ternary group construction process, the diversity of samples is improved, and the decision boundary is accurately learned. In addition, the learned features not only meet the distance measurement constraint, but also meet the matchable discrimination constraint, therefore, a classification loss for judging whether a pair of points belongs to a matchable point pair is introduced in addition to the traditional ternary group loss, and the performance of the learned features for the matching task is further improved. The application not only fully utilizes the distance information between samples, but also can fuse the advantages between features, improve the diversity of samples, and further make the extracted image features have the characteristics of simple calculation and accurate classification. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0045] Figure 1 is a flowchart of the image feature point description method based on sample mixing and improved ternary loss function in the embodiment of the application;

[0046] Figure 2 is a training flowchart of the convolutional neural network for image feature point description in the embodiment of the application;

[0047] Figure 3 is a structure diagram of the convolutional neural network for image feature point description in the embodiment of the application;

[0048] Figure 4 is a sample mixing strategy schematic diagram in the embodiment of the application;

[0049] Figure 5 is a flowchart of the image feature point matching method in the embodiment of the application. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0051] Example 1

[0052] like Figure 1 As shown, the image feature point description method based on sample mixing and an improved ternary loss function according to an embodiment of the present invention includes the following steps:

[0053] S1. Perform key point detection on the given image;

[0054] S2. Extract image blocks of the same size, centered on each key point;

[0055] S3. Input the extracted image patches into a pre-trained convolutional neural network for image feature description (e.g., ...). Figure 3 As shown in the figure, output the feature descriptor vectors of all key points;

[0056] Among them, such as Figure 2 As shown, the training process of this convolutional neural network is as follows:

[0057] Step 1: Construct image samples containing several categories, generating a set of triples (A, P, N). The anchor sample set A includes n anchor samples, the positive sample set P includes n positive samples, and the negative sample set N includes n negative samples. Each anchor sample has a positive sample that forms a matching pair with it.

[0058] Step 2: Input the triples into a pre-constructed convolutional neural network for image feature description to extract the anchor sample feature vector set. and the set of positive sample feature vectors Calculate the distance matrix between positive samples and anchor samples. ;

[0059] Step 3: From the distance matrix Select the two closest and second closest negative sample sets to each anchor sample within a preset threshold range, mix them with the positive sample set in a certain proportion to form the final synthetic negative sample, and use the final synthetic negative sample to replace the negative sample set in the triplet set. Calculate the triplet loss function of the new triplet set. ;

[0060] Step 4: Set the anchor sample feature vectors output by the convolutional neural network The elements in the set are respectively related to the set of positive sample feature vectors. The elements in the negative sample feature vector set The elements in the array are concatenated in order to obtain two sets. and The two are then classified, and the cross-entropy loss function is calculated based on the classification probability. ;

[0061] Step 5: Mix the ternary loss function and the cross-entropy loss function in a certain proportion to obtain the total loss function. Then, use this total loss function to constrain the training process of the convolutional neural network. After multiple training sessions, a convolutional neural network that meets the expectations is obtained.

[0062] As can be seen, the greatest innovation of this embodiment lies in constructing a convolutional neural network for image feature point description. Particularly in the network training phase, by finding the two negative samples closest to the anchor sample and then mixing them with the positive samples using a mixing strategy, the accuracy of sample features is improved while enriching the sample set. Furthermore, binary classification is added during training to further classify the samples, achieving feature fusion. Finally, the ternary loss function and cross-entropy loss function are mixed in a certain proportion as the overall loss function to constrain the model's training process, further improving the accuracy of image feature description.

[0063] Example 2

[0064] This embodiment is based on Embodiment 1, but differs in that it further refines the training process of the convolutional neural network. The innovative approach to network training is as follows: First, initial processing is performed on a batch of samples containing several categories to generate triples. Second, the triples are input into the convolutional neural network to extract feature vectors, and the distance matrix between the anchor sample and the positive sample is calculated. Then, the two closest and second-closest negative samples to the anchor sample are found and mixed with the positive sample in a certain proportion to form the final synthesized negative sample. This negative sample is then used to calculate the triple loss function. Simultaneously, the anchor sample feature vector is concatenated with the positive sample feature vector and the negative sample feature vector, respectively, and the result after passing through a fully connected layer and a softmax activation function is substituted into the cross-entropy loss function for calculation. Finally, the triple loss function and the cross-entropy loss function are added in a certain proportion to obtain the total loss function, which is used to iteratively train the network model until it meets expectations.

[0065] The specific network training process in this embodiment is as follows:

[0066] (1) Prepare the initial network training by taking a batch of samples containing several categories and generate a set of triples (A, P, N);

[0067] (2) Input the above triples into a pre-constructed convolutional neural network for describing image feature points (e.g., Figure 3 (As shown) After that, based on the extracted anchor sample feature vector set and the set of positive sample feature vectors Calculate the distance matrix between positive samples and anchor samples. ;

[0068] (3) Set the threshold range from the matrix Select the two closest and second-closest negative samples to each anchor sample within the threshold range. Combine these two negative samples using a mixing strategy to form an intermediate negative sample. Then, combine the positive sample and the intermediate negative sample using the same mixing strategy to generate the final synthesized negative sample. Finally, use this negative sample to calculate the ternary loss function. ;

[0069] (4) Set of anchor sample feature vectors output by the convolutional neural network The elements in the set are respectively related to the set of positive sample feature vectors. The elements in the negative sample feature vector set The elements in the array are concatenated in order to obtain two sets. and Then, both are input into a fully connected layer and a softmax activation function to calculate the cross-entropy loss function. ;

[0070] (5) Mix the ternary loss function and the cross-entropy loss function in a certain proportion as the total loss function to constrain the training process of the model;

[0071] The above describes the network training part of this invention. Next, I will continue to explain the steps of using this invention in network testing.

[0072] Further, step (1) includes:

[0073] (1.1) For a batch of samples with several categories, select two samples from the same category as anchor samples. and positive samples Select a sample from the class that is not the same as the above categories as the negative sample. Thus, a triplet is formed. Selection complete; after n repetitions of the above operations, a set of triples (A, P, N) is obtained, where sample set A consists of n anchor samples, sample set P consists of n positive samples, and sample set N consists of n negative samples, specifically represented as follows:

[0074] , ,

[0075] However, the selection of negative samples in the triples generated by the above method is too random and not representative, which cannot meet the needs of practical applications. Therefore, the hybrid strategy proposed in this invention can increase the diversity and representativeness of negative samples, thereby extracting features that are simple to calculate and accurate in classification.

[0076] In some alternative implementations, step (2) includes:

[0077] (2.1) After inputting the above triples (A, P, N) into the convolutional neural network, the set of anchor sample feature vectors is extracted. and the set of positive sample feature vectors Thus, the Euclidean distance matrix is ​​calculated. ,in,

[0078] Anchor Sample The feature vector is obtained after passing through a convolutional neural network. ,

[0079] Positive samples The feature vector is obtained after passing through a convolutional neural network. ,

[0080] If and only if i=j and The pairs of objects are matched and retain this matching relationship even after passing through a convolutional neural network.

[0081] ,

[0082]

[0083] (n is the number of samples; m is the number of features)

[0084] In some alternative implementations, step (3) specifically includes:

[0085] (3.1) For each matching pair in a batch of samples From the distance matrix Find its nearest negative sample and the second nearest negative sample The set of most recent negative samples is called the set of most recent negative samples. The set containing the second nearest negative samples is called Meanwhile, threshold Require Each element in the list must satisfy the condition that it is within the specified range, that is, if Then use replace ,in,

[0086] This indicates selecting the second smallest element in the set.

[0087] ,

[0088]

[0089] (3.2) Utilize hybrid strategies (such as...) Figure 4 As shown), first... and Mix them in a certain proportion to obtain intermediate negative samples. Then take the positive sample and intermediate negative samples The final synthetic negative sample is obtained by mixing them in a certain proportion. ,in,

[0090]

[0091]

[0092] in, ,

[0093] The intermediate negative samples should retain more feature information from the closest negative samples, therefore Based on multiple experimental results, it is recommended that the negative samples be mixed in a ratio of 0.9:0.1, meaning that the proportion should be larger. To be optimal;

[0094] To combine the feature information of both positive and negative samples, the final synthesized negative sample is obtained by mixing the positive sample with an intermediate negative sample synthesized through a mixing strategy. The intermediate negative sample should still account for the largest proportion to retain most of the negative sample features, while the positive sample should maintain a small proportion to provide some feature information for learning. Based on the results of multiple experiments, it is recommended to mix the final synthesized negative sample at a ratio of 0.2:0.8. To be optimal;

[0095] (3.3) The above final synthesized negative samples Substitute into the ternary loss function The calculation is performed, and the formula for the ternary loss function is as follows:

[0096]

[0097] Where n represents the number of triples.

[0098] In some alternative implementations, step (4) specifically includes:

[0099] (4.1) The set of anchor sample feature vectors extracted after inputting the set of triples (A, P, N) into the convolutional neural network. Set of positive sample feature vectors and negative sample feature vector set ,in,

[0100] , ,

[0101] negative samples The feature vector is obtained after passing through a convolutional neural network. ,

[0102] The feature vector obtained after passing through a convolutional neural network , , Both are 128-dimensional.

[0103] eigenvectors and , After splicing the two pairs together, we get a 256-dimensional array. , to feature vector and , After splicing the two pairs together, we get a 256-dimensional array. Ultimately, two sets can be obtained. and ,in,

[0104] ,

[0105] (4.2) Label the concatenated feature vectors, where,

[0106] , This represents the concatenation of anchor samples and positive samples. and These are matching pairs, defined as positive, and labeled with label=1.

[0107] , This represents the concatenation of anchor samples and negative samples. and Pairs that do not match are defined as negative classes and labeled with label=0.

[0108] (4.3) Label the feature vector and Two fully connected layers are used to further fuse features, mapping the feature space learned by the convolutional layers to the sample label space. Then, the output of the fully connected layers is transformed by a softmax activation function to obtain the probability that sample i is predicted as positive, denoted as . ;

[0109] (4.4) Finally, the cross-entropy loss function is used. This is used to quantify the distance between the network's output probability and the expected value, thereby improving the network's accuracy.

[0110] Cross-entropy loss function The calculation formula is as follows:

[0111]

[0112] : Represents the actual label of sample i, with 1 for positive class and 0 for negative class.

[0113] : Indicates the probability that sample i is predicted to be of the positive class;

[0114] In some alternative implementations, step (5) specifically includes:

[0115] (5.1) The ternary loss function and the cross-entropy loss function are mixed in a certain proportion to form the total loss function. The formula for calculating the total loss function is as follows:

[0116] ,

[0117] Based on multiple experiments, a higher proportion of the ternary loss function provides positive feedback for network training. Furthermore, a 1:0.8 ratio of ternary loss function to cross-entropy loss function optimizes the training results; therefore, this method is recommended. .

[0118] As can be seen, this embodiment increases the diversity of sample distances on the basis of the original algorithm, and further increases the accuracy of positive and negative sample features by classifying sample feature vectors, which can effectively improve the performance of image feature extraction. Finally, the image features extracted by the feature extraction model of this method perform well in subsequent matching tasks.

[0119] Example 3

[0120] This embodiment is an image feature point matching method. It obtains a more accurate image feature description of the image to be matched by utilizing the image feature description method described in the above embodiment, and then performs image matching, which can improve the accuracy of image matching.

[0121] like Figure 5 As shown, the image feature point matching method in this embodiment includes the following steps:

[0122] Step 1: Obtain the feature descriptor vectors of the two images to be matched using the method described in the above embodiment;

[0123] Step 2: Calculate the distance between key points in the two images based on the feature descriptor vectors;

[0124] Step 3: Match key points based on the distance between them to find key point matching pairs between the two images.

[0125] The distance between the key points is calculated as Euclidean distance.

[0126] Specifically, step 1 includes:

[0127] (1.1) Perform keypoint detection on two given images A and B to obtain N keypoints of image A and M keypoints of image B;

[0128] In some alternative implementations, step 2 specifically includes:

[0129] (2.1) Extract a 65×65 pixel square patch from each of the two images, centered on their respective key points;

[0130] (2.2) After processing and compressing each patch into a size of 32×32 pixels, it is fed into a convolutional neural network to obtain a 128-dimensional feature descriptor;

[0131] In some alternative implementations, step 3 specifically includes:

[0132] (3.1) For each keypoint in image A, calculate the Euclidean distance with each keypoint in image B using its descriptor vector as a similarity metric;

[0133] (3.2) Based on the Euclidean distance between each key point in image A and all key points in image B, a series of matching pairs are found using matching algorithms and optimization strategies to obtain the matching results of the two images.

[0134] Example 4

[0135] This embodiment is based on embodiment 3, and it mainly achieves image matching by using two specific test images.

[0136] In this embodiment, the image matching process mainly includes three aspects: image key point detection, image feature extraction, and image matching, which can be implemented as follows:

[0137] (a) Image keypoint detection: Perform keypoint detection on two given images A and B to obtain N keypoints in image A and M keypoints in image B.

[0138] Image key points can be extracted using algorithms such as SIFT, SURF, SuperPoint, and FAST, but are not limited to these.

[0139] (b) Image Feature Extraction: For both Image A and Image B, a 65×65 pixel square patch is extracted around each extracted keypoint; then, each patch is processed and compressed into a 32×32 pixel grayscale image, and input as follows: Figure 3 In the convolutional neural network shown (i.e. the convolutional neural network for image feature description in the above embodiment), after 6 times of 3×3 convolution and 1 time of 8×8 convolution, and then through BN layer and L2 regularization, a 128-dimensional feature descriptor is obtained. This descriptor vector can be used to describe the local appearance around each key point.

[0140] The set of key points extracted from image A is denoted as . Any keypoint descriptor is denoted as ;

[0141] The set of key points extracted from image B is denoted as . Any keypoint descriptor is denoted as ;

[0142] (c) Image matching: For each key point in image A, the Euclidean distance with each key point in image B is calculated through its descriptor vector. Then, a series of matching pairs are obtained through matching algorithms and optimization strategies, thus obtaining the matching result of the two images.

[0143] In this invention, step (c) can be implemented in the following manner:

[0144] 1) For any keypoint descriptor in image A Calculate the descriptor for each keypoint in image B. The Euclidean distance d is used as a similarity measure, where the Euclidean distance formula is:

[0145] In a two-dimensional plane, Where d is a point With point The distance between them;

[0146] In n-dimensional space, Where d is a point With point The distance between them;

[0147] 2) Based on the Euclidean distance between each keypoint in image A and all keypoints in image B, algorithms such as, but not limited to, Brute Force (BFM), Nearest Neighbor (KNN), and Fast Nearest Neighbor (FLANN) can be used to find a series of matching pairs. The process of finding matching pairs will be illustrated using KNN as an example:

[0148] The Euclidean distance between each keypoint in image A and all keypoints in image B. We retain k best matches, that is, we select k minimum distances, where k is usually 2;

[0149] Set threshold When k is 2, the optimal matching distance is retained. Suboptimal matching distance And apply the threshold to and The ratio is used to filter out fuzzy matches and find the best match for each key point. It is generally 0.8, specifically,

[0150] Only when Only when this is achieved does it indicate that the optimal matching distance has been obtained. The key point in Figure B is significant, so the matching result is retained.

[0151] Using matching algorithms and optimization strategies similar to those described above, a series of matching pairs can be found, thus obtaining the matching results of the two images.

[0152] Table 1 below shows the experimental results of the average accuracy of homography estimation. The results present the accuracy [%] of the model trained by the proposed method and the HardNet++ model at pixel error thresholds of ε = 1, 3, and 5 pixels. All datasets used are the HPatches dataset.

[0153] Table 1. Experimental results of average accuracy of homography estimation

[0154]

[0155] The specific steps are as follows:

[0156] (1) Match the reference image in each test sequence in the HPatches public dataset with the other five images. For each pair of images, extract the matching points using the method proposed in this invention and calculate the corresponding homography matrix.

[0157] (2) To measure the difference between the image after transformation using the above homography matrix and the real homography matrix, we give the accuracy [%] of the image pixel error after the two transformations when the threshold ε is 1, 3 and 5 pixels respectively.

[0158] Experimental results show that the model trained by the method given in this invention is significantly improved compared with the HardNet++ model, demonstrating the significant advantages of this invention in image matching (this invention is based on a series of improvements made to the neural network structure used by HardNet++, hence the comparative experiment between the two).

[0159] It can be seen that this embodiment has the following advantages: (1) This invention not only makes full use of the distance information between samples, but also integrates the advantages between features, improves the diversity of samples, and makes the extracted features simple to calculate and accurate in classification. (2) Under the constraints of this method, the accuracy of feature extraction can be effectively improved, so that the extracted image features can achieve faster matching speed and higher matching accuracy in the matching task, and since the size of the input image is not increased, the training process is simple and fast.

[0160] Example 5

[0161] This embodiment, based on embodiments 1 and 2, is mainly used to implement an image feature point description method. This embodiment's image feature point description system, based on sample mixing and an improved ternary loss function, includes:

[0162] The key point detection module is used to detect key points in a given image;

[0163] The image patch extraction module is used to extract image patches of the same size, centered on each key point.

[0164] The feature description module is used to input the extracted multiple image patches into a pre-trained convolutional neural network for image feature description, and output feature descriptor vectors for all key points; the training process of the convolutional neural network is the training method provided in the above method embodiment.

[0165] Each module is mainly used to implement the various steps of the method embodiment, which will not be elaborated here.

[0166] Example 6

[0167] This embodiment is based on embodiments 3 and 4 and is mainly used to implement an image feature point matching method. The image feature point matching system of this embodiment includes:

[0168] The feature description module is used to obtain feature descriptor vectors for the two images to be matched using the method embodiments described above.

[0169] The distance calculation module is used to calculate the distance between key points in two images based on feature descriptor vectors;

[0170] The matching module is used to find keypoint matching pairs between two images based on the distance between keypoints.

[0171] Each module is mainly used to implement the various steps of the method embodiment, which will not be elaborated here.

[0172] Example 7

[0173] This application also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, app store, etc., which stores a computer program. When the program is executed by a processor, it implements the corresponding function. The image feature point description method based on sample mixing and improved ternary loss function is implemented by the computer-readable storage medium of this embodiment when executed by a processor.

[0174] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.

[0175] The sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0176] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. An image feature point description method based on sample mixing and improved ternary loss function, characterized in that, The method comprises the following steps: S1, key point detection is performed on a given image; S2, image blocks of the same size are respectively extracted with each key point as the center; S3, the extracted multiple image blocks are input into a trained convolutional neural network for image feature description, and a feature descriptor vector of all key points is output; The training process of the convolutional neural network is as follows: Step 1: build an image sample set containing several categories, generate a triple set (A, P, N), the anchor sample set A includes n anchor samples, the positive sample set P includes n positive samples, and the negative sample set N includes n negative samples; each anchor sample has a positive sample forming a matching pair; Step2: input the triple into a pre-constructed convolutional neural network for image feature description, extract the anchor sample feature vector set f a and the positive sample feature vector set f p , calculate the distance matrix Dist_matrix between the positive sample and the anchor sample; Step3: Selecting two negative sample sets closest and second closest to each anchor sample within a preset threshold range from the distance matrix Dist_matrix, mixing them with the positive sample set according to a certain proportion to form a final synthetic negative sample set, and replacing the negative sample set in the triple set with the final synthetic negative sample set to calculate the triple loss function L Triplet ; Step4: concatenate the elements of the anchor sample feature vector set f a outputted by the convolutional neural network with the elements of the positive sample feature vector set f p and the elements of the negative sample feature vector set f n in sequence to obtain two sets out ap and out an , classify them, and calculate the cross-entropy loss function L CE according to the classification probability; Step 5: mix the triple loss function and the cross-entropy loss function according to a certain proportion as a total loss function, and then constrain the training process of the convolutional neural network according to the total loss function, and obtain the expected convolutional neural network after multiple training.

2. The image feature point description method based on sample mixing and improved ternary loss function according to claim 1, characterized in that, Step 4 further comprises: The feature vectors obtained after splicing are labeled, wherein the feature vectors spliced from the anchor sample and the positive sample are defined as positive classes, and are labeled as label=1; the feature vectors spliced from the anchor sample and the negative sample are defined as negative classes, and are labeled as label=0; The two sets out ap and out an are labeled in succession through two fully connected layers to realize further fusion of features; The output of the fully connected layer is converted by a softmax activation function to obtain the probability that sample i is predicted as a positive class, denoted as p i ; The cross-entropy loss function L is used to quantify the distance between the output probability of the network and the expected value CE The cross-entropy loss function L CE is calculated as follows: y i y represents the actual label of sample i, positive class is 1, negative class is 0, p i y represents the probability that sample i is predicted as a positive class.

3. The method of claim 1, wherein the method is based on sample mixing and improved ternary loss function for image feature point description. Step 3 is specifically: Selecting two negative sample sets closest and second closest to each anchor sample within a preset threshold range from the distance matrix Dist_matrix, mixing the two negative sample sets according to a certain ratio for the first time to form an intermediate negative sample set, and mixing the positive sample set and the intermediate negative sample set according to a ratio for the second time to generate a final synthetic negative sample set, replacing the negative sample set N in the original triple set (A, P, N), and calculating the triple loss function L Triplet .

4. The image feature point description method based on sample mixing and improved ternary loss function according to claim 3, characterized in that, When the two negative sample sets are mixed according to a certain proportion in Step 3, the mixing process is mix_neg=λmin_neg+(1-λ)min_sec_neg, wherein mix_neg is the intermediate negative sample set, min_neg is the closest negative sample set, and min_sec_neg is the second closest negative sample set, and λ∈(0.9,1).

5. The image feature point description method based on sample mixing and improved ternary loss function according to claim 4, characterized in that, In the second mixing of the positive sample set and the intermediate negative sample set in Step Step3, the mixing process is final_neg = apos + (1-a)mix_neg, where final_neg is the final synthetic negative sample set, pos = [d(a1, p1), d(a2, p2)…d(a n ,p n )] and mix_neg = [d(a i ,p i )] are the distances between each matched pair, and a e (0, 0.2).

6. The method of claim 1, wherein, In Step 5, the triple loss function and the cross-entropy loss function are mixed according to a certain proportion as a total loss function Loss, and the specific mixing formula is as follows: Loss = L Triplet + ωL CE , ω e (0.5, 1].

7. An image feature point matching method characterized by, The method comprises the following steps: Step 1: obtaining feature descriptor vectors of two images to be matched by the method in any one of claims 1-5; Step 2: calculating the distance between key points in the two images according to the feature descriptor vectors; Step 3: matching according to the distance between the key points to find the key point matching pairs of the two images.

8. The image feature point matching method according to claim 7, wherein The calculated distance between the key points is the Euclidean distance.

9. An image feature point description system based on sample mixing and improved ternary loss function, characterized in that, It comprises: a key point detection module for performing key point detection on a given image; an image block extraction module for extracting image blocks of the same size with each key point as the center; a feature description module for inputting the extracted multiple image blocks into a trained convolutional neural network for image feature description, and outputting a feature descriptor vector of all key points; the training process of the convolutional neural network is the steps Step1-Step5 in any one of claims 1-6.

10. An image feature point matching system, characterized by, It comprises: a feature description module for obtaining feature descriptor vectors of two images to be matched by the method in any one of claims 1-6; a distance calculation module for calculating the distance between key points in the two images according to the feature descriptor vectors; A matching module is configured to match the key points according to the distance between the key points to find the matching pairs of the key points of the two images.

Images