A feature mismatch detection method combining image semantics and coordinate relations

Abstract

The present invention discloses a feature mismatch detection method combining image semantics and coordinate relationship. The method detects image mismatch points by a deep learning detection network composed of three specific functional modules. The three functional modules include a semantic vector generation module, a coordinate position relationship vector generation module, and a matching point pair discrimination module. The input of the semantic vector generation module needs to be aligned in rotation direction and scale first. The deep learning network is trained on a large number of labeled training data sets containing correct matching point pairs and mismatching point pairs. After training, the network can effectively discriminate the correctness of the matching relationship of the matching image point pairs to be detected. The present invention adopts a new deep learning method to significantly improve the accuracy and robustness of the discrimination of the correctness of matching point pairs. In addition, the present invention can also provide a high-quality feature point matching relationship verification method for applications in the fields of face recognition, three-dimensional reconstruction, image retrieval, and autonomous driving.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to a feature mismatch detection method combining image semantics and coordinate relationship. Background Art

[0002] Image feature point matching technology has a wide range of applications in computer vision tasks, such as face recognition, 3D reconstruction, image retrieval, and autonomous driving. At present, there are many classic algorithms in the field of feature point matching, including SIFT, SURF, FAST, HoG, CONTEXTDESC, etc. Although these classic algorithms have taken into account the differences in rotation, scale, and illumination changes between the images to be matched, when dealing with complex scenes, the above methods still cannot avoid the problem of false matching, hereinafter referred to as false matching. Therefore, the research work on false matching detection aims to improve the matching accuracy by eliminating false matching feature point pairs in the matching results. Existing false matching detection methods mainly include two categories: false matching detection methods that use matching relationship statistics and machine vision imaging modeling theory, such as the RANSAC algorithm. This method uses an iterative optimization method to calculate the basic matrix that meets the feature point matching, thereby eliminating external points that do not meet the basic matrix, namely false matching points. Mismatch detection methods using deep learning methods, such as the NM-NET algorithm, use the coordinates of all matching point pairs of the two images to be matched as network input, and use neural networks to learn the features of mismatched point pairs, so as to judge whether the matching result point pairs are correct or not. Its essence is still to detect outliers in the coordinate relationship of matching point pairs. In recent years, high-precision detection of mismatched point pairs is still a hot issue in the field of computer vision and image processing. For data sets with large changes in viewing angles, scales, and lighting conditions, the relevant algorithms still have bottlenecks such as low mismatch detection accuracy and low algorithm robustness. Different from the above methods, this patent proposes a new feature mismatch detection method, which not only relies on the coordinate relationship of the matching feature point pairs to be detected, but also combines the image semantic information of the matching feature point pairs, and constructs a deep learning network structure different from the existing methods to achieve higher-precision mismatch feature point detection. Summary of the invention

[0003] The purpose of the present invention is to provide a feature mismatch detection method that combines image semantics and coordinate relationships. Compared with existing feature mismatch detection methods, the method can significantly improve the detection accuracy of mismatch feature points.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is:

[0005] A feature mismatch detection method combining image semantics and coordinate relationship comprises the following steps:

[0006] Step 1: select a large number of images to be matched, use an automated matching calculation method plus a manual interactive annotation method to obtain a large number of correct matching point pairs and incorrect matching point pairs, and select 2000 matching point pairs from all the matching point pairs as a sample. The sample data constitutes the training data set;

[0007] Step 2: Build a multimodal neural network structure that can process neighborhood image blocks and image coordinates. The neural network structure mainly includes three modules: semantic vector generation module, coordinate position relationship vector generation module, and matching point pair discrimination module.

[0008] Step 3: For each matching point pair in the training data set, the neighborhood image blocks of the matching point pair are aligned and preprocessed by using scale normalization and image rotation operations, and the aligned neighborhood image blocks of the matching point pair are input into the semantic vector generation module built in step 2 to obtain the semantic vectors of all matching point pairs;

[0009] Step 4, obtain the position coordinates of all matching point pairs of each sample in the training data set, and input the position coordinates of all matching point pairs of each sample into the coordinate position relationship vector generation module built in step 2, obtain the position information feature vector of all matching point pairs of each sample, and perform feature fusion on the neighborhood image semantic vector and the position information feature vector of all matching point pairs of each sample;

[0010] Step 5, construct a parameterized neural network model, input the neighborhood image blocks and position coordinates of all matching point pairs of each sample in the training data set into the parameterized neural network model, and obtain the output result;

[0011] Step 6: Use the fully connected layer and the cross entropy loss function to perform binary classification on the fused feature vector of step 4 and the output result of step 5, and back-propagate to update the weight parameters of the multimodal neural network built in step 2 and the parameterized neural network built in step 5. Use the training data set data to train the two networks respectively, and save two different sets of weight bias parameters.

[0012] Step 7, obtain the neighborhood image block and position coordinates of the sample to be tested, input them into the multimodal neural network and parameterized neural network trained in step 6 respectively, and obtain the prediction result of the sample to be tested.

[0013] Furthermore, the specific method of step 1 is as follows:

[0014] Step 1.1, select two original images to be matched, use the automatic matching algorithm to match the two original images, and obtain several matching point pairs of the images, each matching point pair consists of two feature points on the original image;

[0015] Step 1.2: According to the result of step 1.1, if the two feature points corresponding to each matching point pair represent the same position in the real scene, the label of this matching point pair is a correct match; if they are not the same position, the label of this matching point pair is an incorrect match. Similarly, the labels of all matching point pairs are obtained, and 2000 matching point pairs are selected from all matching point pairs as a sample.

[0016] Step 1.3, repeat steps 1.1 to 1.2 to create 1500 samples as training data sets, where each sample corresponds to two original images, 2000 matching point pairs, and each matching point pair corresponds to two feature points.

[0017] Furthermore, the specific method of step 2 is as follows:

[0018] Step 2.1, build a semantic vector generation module with a total of 14 layers, namely convolution layer 1, BN layer 1, convolution layer 2, BN layer 2, convolution layer 3, BN layer 3, convolution layer 4, BN layer 4, convolution layer 5, BN layer 5, convolution layer 6, BN layer 6, convolution layer 7, BN layer 7, where the output of each layer is the input of the next layer;

[0019] Step 2.2, build a coordinate position relationship vector generation module with a total of 18 layers, namely convolution layer 1, IN layer 1, BN layer 1, convolution layer 2, IN layer 2, BN layer 2, convolution layer 3, IN layer 3, BN layer 3, convolution layer 4, IN layer 4, BN layer 4, convolution layer 5, IN layer 5, BN layer 5, convolution layer 6, IN layer 6, BN layer 6, where the output of each layer is the input of the next layer;

[0020] In step 2.3, a matching point pair discrimination module with a total of 5 layers is built, namely fully connected layer 1, fully connected layer 2, fully connected layer 3, fully connected layer 4, and fully connected layer 5, where the output of each layer is the input of the next layer.

[0021] Furthermore, the specific method of step 3 is as follows:

[0022] Step 3.1, select a sample in the training data set, each sample has 2000 matching point pairs, select one of the matching point pairs, each matching point pair corresponds to the left and right feature points, find the position coordinates of these two feature points on the original image in step 1.1, take the position coordinates as the center, select a range of 32 pixels in length and 32 pixels in width in the main direction of the point, segment the original image, and use the segmented image as the neighborhood image block of the feature point. Repeat the above operation to obtain the neighborhood image block of another feature point;

[0023] Step 3.2, according to the result of step 3.1, scale normalization is performed on the two neighborhood image blocks of the matching point pair, and the two neighborhood image blocks are respectively input into the semantic vector generation module constructed in step 2.1 to obtain feature vectors, and the two feature vectors are normalized and added together as the semantic vector of the matching point pair;

[0024] Step 3.3, repeat steps 3.1 to 3.2 to obtain the semantic vectors of all matching point pairs in the training set.

[0025] Furthermore, the specific method of step 4 is as follows:

[0026] Step 4.1, select a sample in the training data set, select a matching point pair in the sample, obtain the position coordinates of the two feature points corresponding to the matching point pair on the original image in step 1.1, and horizontally splice and normalize the two position coordinates. Repeat the above operations to obtain the position coordinates of all matching point pairs of the sample, and vertically splice the position coordinates of all matching point pairs of the sample together to form a 2000*4 whole;

[0027] Step 4.2, according to the result of step 4.1, the position coordinates of all matching point pairs of the sample are input into the coordinate position relationship vector generation module constructed in step 2.2, and the coordinate position relationship vectors of all matching point pairs can be obtained;

[0028] Step 4.3, according to the results of step 4.2 and step 3.2, the coordinate position relationship vectors of all matching point pairs of the sample are normalized and added to the semantic vector to obtain the fused feature vectors of all matching point pairs of the sample;

[0029] Step 4.4, repeat steps 4.1 to 4.3 to obtain the fused feature vector of all samples in the training data set.

[0030] Furthermore, the specific method of step 5 is as follows:

[0031] Construct a parameterized neural network model, which mainly includes three modules: weight parameter generation module, coordinate position relationship vector generation module, and matching point pair discrimination module;

[0032] Step 5.1, build a weight parameter generation module with a total of 12 layers, in which convolutional layers and BN layers are used alternately, namely convolutional layer 1, BN layer 1, convolutional layer 2, BN layer 2, convolutional layer 3, BN layer 3, convolutional layer 4, BN layer 4, convolutional layer 5, BN layer 5, convolutional layer 6, BN layer 6, where the output of each layer is the input of the next layer;

[0033] Step 5.2, build a coordinate position relationship vector generation module with a total of 18 layers, in which convolutional layers, IN layers, and BN layers are used alternately, namely convolutional layer 1, IN layer 1, BN layer 1, convolutional layer 2, IN layer 2, BN layer 2, convolutional layer 3, IN layer 3, BN layer 3, convolutional layer 4, IN layer 4, BN layer 4, convolutional layer 5, IN layer 5, BN layer 5, convolutional layer 6, IN layer 6, BN layer 6, where the output of each layer is the input of the next layer, and the weight parameters of convolutional layer 2 and convolutional layer 4 are the outputs of the weight parameter generation module built in step 5.1;

[0034] Step 5.3, build a matching point pair discrimination module with a total of 5 layers, namely fully connected layer 1, fully connected layer 2, fully connected layer 3, fully connected layer 4, and fully connected layer 5, where the output of each layer is the input of the next layer;

[0035] Step 5.4, according to the results of step 3 and step 4, a sample is selected from the training data set, and the position coordinates of all matching point pairs of the sample and the neighborhood image blocks are respectively input into the coordinate position relationship vector generation module and the weight parameter generation module in the parameterized neural network constructed in step 5 to obtain the output result of the parameterized neural network;

[0036] Step 5.5, repeat the above steps 5.1-5.4 to obtain the output results of all samples.

[0037] Furthermore, the specific method of step 6 is as follows:

[0038] Step 6.1, according to the result of step 4.4, select a sample in the training data set, input the fusion feature vector of the sample into the matching point pair discrimination module constructed in step 2.3 and perform softmax calculation on the output result. The softmax formula is shown in formula (1), where i represents the output value of the i-th node in the last layer of the neural network, e represents the exponential function, and the last layer of the neural network has two nodes, which correspond to the two situations of correct matching and incorrect matching respectively. S i The value of represents the probability that the match is a correct match or an incorrect match. The calculation result of softmax is used as the input of the cross entropy loss function, and the output value of the cross entropy loss function is calculated. This output value is the loss value. The formula of the cross entropy loss function is shown in formula (2), where p(x) is the label value, a label value of 1 represents a correct match, and a label value of 0 represents an incorrect match. q(x) is the probability that the match is a correct match or an incorrect match, and H(X, q) represents the loss value. According to the loss value, back propagation calculation is performed, and the weight bias parameters in the multimodal neural network built in step 2 are updated;

[0039]

[0040] H(X,q)=-∑p(x)logq(x) (2);

[0041] Step 6.2, repeat step 6.1, use all the sample data in the training data set to train the multimodal neural network built in step 2 for 15 epochs, and then train the coordinate position relationship vector generation module in the network separately for 100 epochs, and adjust the learning rate parameter of the multimodal neural network according to the loss curve so that the loss curve tends to converge. After the training is completed, the weight bias parameters of the multimodal neural network are exported and saved;

[0042] Step 6.3, according to the result of step 5.4, select a sample in the training data set, input the parameterized neural network output result of the sample into the matching point pair discrimination module built in step 5.3 and perform softmax calculation on the output result, use the softmax calculation result as the input of the cross entropy loss function, calculate the output value of the cross entropy loss function, and this output value is the loss value. According to the loss value, perform back propagation calculation and update the weight bias parameters in the parameterized neural network built in step 5;

[0043] Step 6.4, repeat step 5.4, use all the sample data in the training data set to train the parameterized neural network built in step 5 for 15 epochs, and adjust the learning rate parameter of the parameterized neural network according to the loss curve so that the loss curve tends to converge. After the training is completed, the weight bias parameters of the parameterized neural network are exported and saved.

[0044] Furthermore, the specific method of step 7 is as follows:

[0045] Step 7.1, select a sample to be tested, and obtain the neighborhood image block and position coordinates of the sample;

[0046] Step 7.2, according to the result of step 6.2, the weight parameters are imported into the multimodal neural network constructed in step 2, the result of step 7.1 is input into the multimodal neural network, and the output result is subjected to softmax operation, and the label corresponding to the largest value is selected as the prediction result of the network;

[0047] Step 7.3, according to the result of step 6.4, the weight parameters are imported into the parameterized neural network constructed in step 5, the result of step 7.1 is input into the parameterized neural network, and the output result is subjected to softmax operation, and the label corresponding to the largest value is selected as the prediction result of the network;

[0048] Step 7.4, according to the result obtained in step 7.2, if the predicted label is right, then this match is a correct match; if the predicted label is wrong, then this match is an incorrect match;

[0049] Step 7.5, repeat steps 7.1, 7.2, and 7.4 to obtain the prediction results of all the samples to be tested of the network built in step 2;

[0050] Step 7.6, repeat steps 7.1, 7.3, and 7.4 to obtain the prediction results of all the test samples of the parameterized neural network constructed in step 5.

[0051] The beneficial effects of the present invention are:

[0052] The present invention adopts deep learning theory to construct a new network structure suitable for mismatch detection, which can simultaneously process image semantic information and coordinate information of matching point sets. Through network training on a large number of feature point matching data sets, the network can accurately distinguish input mismatch feature point pairs. Compared with the existing mismatch feature detection method, the present invention can greatly improve the accuracy of mismatch feature point detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Attached Figure 1 This is a flow chart of a feature mismatch detection method combining image semantics and coordinate relationships according to the present invention. DETAILED DESCRIPTION

[0054] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] A feature mismatch detection method combining image semantics and coordinate relationship is implemented in the following steps:

[0056] Step 1: Select a large number of images to be matched, use the automatic matching calculation method plus the manual interactive annotation method to obtain a large number of correct matching point pairs and mismatching point pairs, and select 2000 matching point pairs from all the matching point pairs as a sample. The sample data constitutes the training data set.

[0057] The specific operations of step 1 are:

[0058] Step 1.1, select two original images to be matched, use the automatic matching algorithm to match the two original images, and obtain several matching point pairs of the images, each matching point pair consists of two feature points on the original image;

[0059] Step 1.2: According to the result of step 1.1, if the two feature points corresponding to each matching point pair represent the same position in the real scene, the label of this matching point pair is a correct match; if they are not the same position, the label of this matching point pair is an incorrect match. Similarly, the labels of all matching point pairs are obtained, and 2000 matching point pairs are selected from all matching point pairs as a sample.

[0060] Step 1.3, repeat steps 1.1 to 1.2 to create 1500 samples as training data sets, where each sample corresponds to two original images, 2000 matching point pairs, and each matching point pair corresponds to two feature points.

[0061] Step 2: Build a multimodal neural network structure that can process neighborhood image blocks and image coordinates. The neural network structure mainly includes three modules: semantic vector generation module, coordinate position relationship vector generation module, and matching point pair discrimination module.

[0062] The specific operations of step 2 are:

[0063] Step 2.1, build a semantic vector generation module with a total of 14 layers, namely convolution layer 1, BN layer 1, convolution layer 2, BN layer 2, convolution layer 3, BN layer 3, convolution layer 4, BN layer 4, convolution layer 5, BN layer 5, convolution layer 6, BN layer 6, convolution layer 7, BN layer 7, where the output of each layer is the input of the next layer, and the parameter settings are shown in Table 1;

[0064] Table 1 Specific design parameters of semantic vector generation module

[0065] Layer number Network Type Input size Convolution Kernel Step Length filling Output size 1 Convolution 1*1*32*32 3*3 1 1 1*32*32*32 2 BN 1*32*32*32 - - - 1*32*32*32 3 Convolution 1*32*32*32 3*3 1 1 1*32*32*32 4 BN 1*32*32*32 - - - 1*32*32*32 5 Convolution 1*32*32*32 3*3 2 1 1*64*16*16 6 BN 1*64*16*16 - - - 1*64*16*16 7 Convolution 1*64*16*16 3*3 1 1 1*64*16*16 8 BN 1*64*16*16 - - - 1*64*16*16 9 Convolution 1*64*16*16 3*3 2 1 1*128*8*8 10 BN 1*128*8*8 - - - 1*128*8*8 11 Convolution 1*128*8*8 3*3 1 1 1*128*8*8 12 BN 1*128*8*8 - - - 1*128*8*8 13 Convolution 1*128*8*8 8*8 1 0 1*128*1*1 14 BN 1*128*1*1 - - - 1*128*1*1

[0066] Step 2.2, build a coordinate position relationship vector generation module with a total of 18 layers, namely convolution layer 1, IN layer 1, BN layer 1, convolution layer 2, IN layer 2, BN layer 2, convolution layer 3, IN layer 3, BN layer 3, convolution layer 4, IN layer 4, BN layer 4, convolution layer 5, IN layer 5, BN layer 5, convolution layer 6, IN layer 6, BN layer 6, where the output of each layer is the input of the next layer, and the parameter settings are shown in Table 2;

[0067] Table 2 Specific design parameters of the coordinate position relationship vector generation module

[0068]

[0069]

[0070] Step 2.3, build a matching point pair discrimination module with a total of 5 layers, namely fully connected layer 1, fully connected layer 2, fully connected layer 3, fully connected layer 4, and fully connected layer 5. The output of each layer is the input of the next layer. The parameter settings are shown in Table 3.

[0071] Table 3 Specific design parameters of the matching point pair discrimination module

[0072] Layer number Network Type Input size Output size 1 FullyConnected 1*128 1*64 2 FullyConnected 1*64 1*32 3 FullyConnected 1*32 1*16 4 FullyConnected 1*16 1*8 5 FullyConnected 1*8 1*2

[0073] Step 3: For each matching point pair in the training data set, the neighborhood image blocks of the matching point pair are aligned and preprocessed by using scale normalization and image rotation operations, and the aligned neighborhood image blocks of the matching point pair are input into the semantic vector generation module built in step 2 to obtain the semantic vectors of all matching point pairs;

[0074] The specific operations of step 3 are:

[0075] Step 3.1, select a sample in the training data set, each sample has 2000 matching point pairs, select one of the matching point pairs, each matching point pair corresponds to the left and right feature points, find the position coordinates of these two feature points on the original image in step 1.1, take the position coordinates as the center, select a range of 32 pixels in length and 32 pixels in width in the main direction of the point, segment the original image, and use the segmented image as the neighborhood image block of the feature point. Repeat the above operation to obtain the neighborhood image block of another feature point;

[0076] Step 3.2, according to the result of step 3.1, scale normalization is performed on the two neighborhood image blocks of the matching point pair, and the two neighborhood image blocks are respectively input into the semantic vector generation module constructed in step 2.1 to obtain feature vectors, and the two feature vectors are normalized and added together as the semantic vector of the matching point pair;

[0077] Step 3.3, repeat steps 3.1 to 3.2 to obtain the semantic vectors of all matching point pairs in the training set.

[0078] Step 4, obtain the position coordinates of all matching point pairs of each sample in the training data set, and input the position coordinates of all matching point pairs of each sample into the coordinate position relationship vector generation module built in step 2, obtain the position information feature vector of all matching point pairs of each sample, and perform feature fusion on the neighborhood image semantic vector and the position information feature vector of all matching point pairs of each sample;

[0079] The specific operations of step 4 are:

[0080] Step 4.1, select a sample in the training data set, select a matching point pair in the sample, obtain the position coordinates of the two feature points corresponding to the matching point pair on the original image in step 1.1, and horizontally splice and normalize the two position coordinates. Repeat the above operations to obtain the position coordinates of all matching point pairs of the sample, and vertically splice the position coordinates of all matching point pairs of the sample together to form a 2000*4 whole;

[0081] Step 4.2, according to the result of step 4.1, the position coordinates of all matching point pairs of the sample are input into the coordinate position relationship vector generation module constructed in step 2.2, and the coordinate position relationship vectors of all matching point pairs can be obtained;

[0082] Step 4.3, according to the results of step 4.2 and step 3.2, the coordinate position relationship vectors of all matching point pairs of the sample are normalized and added to the semantic vector to obtain the fused feature vectors of all matching point pairs of the sample;

[0083] Step 4.4, repeat steps 4.1 to 4.3 to obtain the fused feature vector of all samples in the training data set.

[0084] In step 5, a parameterized neural network model is constructed. The network mainly includes three modules, namely, a weight parameter generation module, a coordinate position relationship vector generation module, and a matching point pair discrimination module. The neighborhood image blocks and position coordinates of all matching point pairs of each sample in the training data set are input into the parameterized neural network model to obtain the output result;

[0085] The specific operations of step 5 are:

[0086] Step 5.1, build a weight parameter generation module with a total of 12 layers, in which the convolution layer and the BN layer are used alternately, which are convolution layer 1, BN layer 1, convolution layer 2, BN layer 2, convolution layer 3, BN layer 3, convolution layer 4, BN layer 4, convolution layer 5, BN layer 5, convolution layer 6, BN layer 6, in which the output of each layer is the input of the next layer, and the parameter settings are shown in Table 4;

[0087] Table 4 Specific design parameters of weight parameter generation module

[0088] Layer number Network Type Input size Convolution Kernel Step Length filling Output size 1 Convolution 1*1*32*32 3*3 1 1 1*32*32*32 2 BN 1*32*32*32 - - - 1*32*32*32 3 Convolution 1*32*32*32 3*3 1 1 1*32*32*32 4 BN 1*32*32*32 - - - 1*32*32*32 5 Convolution 1*32*32*32 3*3 2 1 1*64*16*16 6 BN 1*64*16*16 - - - 1*64*16*16 7 Convolution 1*64*16*16 3*3 1 1 1*64*16*16 8 BN 1*64*16*16 - - - 1*64*16*16 9 Convolution 1*64*16*16 3*3 2 1 1*128*8*8 10 BN 1*128*8*8 - - - 1*128*8*8 11 Convolution 1*128*8*8 3*3 1 1 1*128*8*8 12 BN 1*128*8*8 - - - 1*128*8*8

[0089] Step 5.2, build a coordinate position relationship vector generation module with a total of 18 layers, in which convolutional layers, IN layers, and BN layers are used alternately, namely convolutional layer 1, IN layer 1, BN layer 1, convolutional layer 2, IN layer 2, BN layer 2, convolutional layer 3, IN layer 3, BN layer 3, convolutional layer 4, IN layer 4, BN layer 4, convolutional layer 5, IN layer 5, BN layer 5, convolutional layer 6, IN layer 6, BN layer 6, where the output of each layer is the input of the next layer, and the weight parameters of convolutional layer 2 and convolutional layer 4 are the outputs of the weight parameter generation module built in step 5.1. The network parameter settings are shown in Table 5;

[0090] Table 5 Specific design parameters of the coordinate position relationship vector generation module

[0091]

[0092]

[0093] Step 5.3, build a matching point pair discrimination module with a total of 5 layers, namely fully connected layer 1, fully connected layer 2, fully connected layer 3, fully connected layer 4, and fully connected layer 5. The output of each layer is the input of the next layer. The specific operation is the same as step 2.3;

[0094] Step 5.4, according to the results of step 3 and step 4, a sample is selected from the training data set, and the position coordinates of all matching point pairs of the sample and the neighborhood image blocks are respectively input into the coordinate position relationship vector generation module and the weight parameter generation module in the parameterized neural network constructed in step 5 to obtain the output result of the parameterized neural network;

[0095] Step 5.5, repeat the above steps to obtain the output results of all samples.

[0096] Step 6: Use the fully connected layer and the cross entropy loss function to perform binary classification on the fused feature vector of step 4 and the output result of step 5, and back-propagate to update the weight parameters of the multimodal neural network built in step 2 and the parameterized neural network built in step 5. Use the training data set data to train the two networks respectively, and save two different sets of weight bias parameters.

[0097] The specific operations of step 6 are:

[0098] Step 6.1, according to the result of step 4.4, select a sample in the training data set, input the fusion feature vector of the sample into the matching point pair discrimination module constructed in step 2.3 and perform softmax calculation on the output result. The softmax formula is shown in formula (1), where i represents the output value of the i-th node in the last layer of the neural network, e represents the exponential function, and the last layer of the neural network has two nodes, which correspond to the two situations of correct matching and incorrect matching respectively. S i The value of represents the probability that the match is a correct match or an incorrect match. The calculation result of softmax is used as the input of the cross entropy loss function, and the output value of the cross entropy loss function is calculated. This output value is the loss value. The formula of the cross entropy loss function is shown in formula (2), where p(x) is the label value, a label value of 1 represents a correct match, and a label value of 0 represents an incorrect match. q(x) is the probability that the match is a correct match or an incorrect match, and H(X, q) represents the loss value. According to the loss value, back propagation calculation is performed, and the weight bias parameters in the multimodal neural network built in step 2 are updated;

[0099]

[0100] H(X,q)=-∑p(x)logq(x) (2);

[0101] Step 6.2, repeat step 6.1, use all the sample data in the training data set to train the multimodal neural network built in step 2 for 15 epochs, and then train the coordinate position relationship vector generation module in the network separately for 100 epochs, and adjust the learning rate parameter of the multimodal neural network according to the loss curve so that the loss curve tends to converge. After the training is completed, the weight bias parameters of the multimodal neural network are exported and saved;

[0102] Step 6.3, according to the result of step 5.4, select a sample in the training data set, input the parameterized neural network output result of the sample into the matching point pair discrimination module built in step 5.3 and perform softmax calculation on the output result, use the softmax calculation result as the input of the cross entropy loss function, calculate the output value of the cross entropy loss function, and this output value is the loss value. According to the loss value, perform back propagation calculation and update the weight bias parameters in the parameterized neural network built in step 5;

[0103] Step 6.4, repeat step 5.4, use all the sample data in the training data set to train the parameterized neural network built in step 5 for 15 epochs, and adjust the learning rate parameter of the parameterized neural network according to the loss curve so that the loss curve tends to converge. After the training is completed, the weight bias parameters of the parameterized neural network are exported and saved.

[0104] Step 7, obtaining the neighborhood image block and position coordinates of the sample to be tested, and inputting them into the multimodal neural network and parameterized neural network trained in step 6 respectively, and obtaining the prediction result of the sample to be tested;

[0105] The specific operations of step 7 are:

[0106] Step 7.1, select a sample to be tested, and obtain the neighborhood image block and position coordinates of the sample;

[0107] Step 7.2, according to the result of step 6.2, the weight parameters are imported into the multimodal neural network constructed in step 2, the result of step 7.1 is input into the multimodal neural network, and the output result is subjected to softmax operation, and the label corresponding to the largest value is selected as the prediction result of the network;

[0108] Step 7.3, according to the result of step 6.4, the weight parameters are imported into the parameterized neural network constructed in step 5, the result of step 7.1 is input into the parameterized neural network, and the output result is subjected to softmax operation, and the label corresponding to the largest value is selected as the prediction result of the network;

[0109] Step 7.4, according to the result obtained in step 7.2, if the predicted label is right, then this match is a correct match; if the predicted label is wrong, then this match is an incorrect match;

[0110] Step 7.5, repeat steps 7.1, 7.2, and 7.4 to obtain the prediction results of all the samples to be tested of the network built in step 2;

[0111] Step 7.6, repeat steps 7.1, 7.3, and 7.4 to obtain the prediction results of all the test samples of the parameterized neural network constructed in step 5.

[0112] The above-mentioned embodiments only express the specific implementation of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A feature mismatch detection method combining image semantics and coordinate relationships, characterized in that: The following steps are involved: Step 1: select a large number of images to be matched, use an automated matching calculation method plus a manual interactive annotation method to obtain a large number of correct matching point pairs and incorrect matching point pairs, and select 2000 matching point pairs from all the matching point pairs as a sample. The sample data constitutes the training data set; Step 2: Build a multimodal neural network structure that can process neighborhood image blocks and image coordinates. The neural network structure includes three modules: a semantic vector generation module, a coordinate position relationship vector generation module, and a matching point pair discrimination module. Step 3: For each matching point pair in the training data set, the neighborhood image blocks of the matching point pair are aligned and preprocessed by using scale normalization and image rotation operations, and the aligned neighborhood image blocks of the matching point pair are input into the semantic vector generation module built in step 2 to obtain the semantic vectors of all matching point pairs; Step 4, obtain the position coordinates of all matching point pairs of each sample in the training data set, and input the position coordinates of all matching point pairs of each sample into the coordinate position relationship vector generation module built in step 2, obtain the position information feature vector of all matching point pairs of each sample, and perform feature fusion on the neighborhood image semantic vector and the position information feature vector of all matching point pairs of each sample; Step 5, construct a parameterized neural network model, input the neighborhood image blocks and position coordinates of all matching point pairs of each sample in the training data set into the parameterized neural network model, and obtain the output result; Step 6: Use the fully connected layer and the cross entropy loss function to perform binary classification on the fused feature vector of step 4 and the output result of step 5, and back-propagate to update the weight parameters of the multimodal neural network built in step 2 and the parameterized neural network built in step 5. Use the training data set data to train the two networks respectively, and save two different sets of weight bias parameters. Step 7, obtain the neighborhood image block and position coordinates of the sample to be tested, input them into the multimodal neural network and parameterized neural network trained in step 6 respectively, and obtain the prediction result of the sample to be tested.

2. The feature mismatch detection method combining image semantics and coordinate relationship according to claim 1, characterized in that: The specific steps of step 1 are as follows: Step 1.1, select two original images to be matched, use the automatic matching algorithm to match the two original images, and obtain several matching point pairs of the images, each matching point pair consists of two feature points on the original image; Step 1.2: According to the result of step 1.1, if the two feature points corresponding to each matching point pair represent the same position in the real scene, the label of this matching point pair is a correct match; if they are not the same position, the label of this matching point pair is an incorrect match. Similarly, the labels of all matching point pairs are obtained, and 2000 matching point pairs are selected from all matching point pairs as a sample. Step 1.3, repeat steps 1.1 to 1.2 to create 1500 samples as training data sets, where each sample corresponds to two original images, 2000 matching point pairs, and each matching point pair corresponds to two feature points.

3. The feature mismatch detection method combining image semantics and coordinate relationship according to claim 2, characterized in that: The specific steps of step 2 are as follows: Step 2.1, build a semantic vector generation module with a total of 14 layers, namely convolution layer 1, BN layer 1, convolution layer 2, BN layer 2, convolution layer 3, BN layer 3, convolution layer 4, BN layer 4, convolution layer 5, BN layer 5, convolution layer 6, BN layer 6, convolution layer 7, BN layer 7, where the output of each layer is the input of the next layer; Step 2.2, build a coordinate position relationship vector generation module with a total of 18 layers, namely convolution layer 1, IN layer 1, BN layer 1, convolution layer 2, IN layer 2, BN layer 2, convolution layer 3, IN layer 3, BN layer 3, convolution layer 4, IN layer 4, BN layer 4, convolution layer 5, IN layer 5, BN layer 5, convolution layer 6, IN layer 6, BN layer 6, where the output of each layer is the input of the next layer; In step 2.3, a matching point pair discrimination module with a total of 5 layers is built, namely fully connected layer 1, fully connected layer 2, fully connected layer 3, fully connected layer 4, and fully connected layer 5, where the output of each layer is the input of the next layer.

4. The feature mismatch detection method combining image semantics and coordinate relationship according to claim 3, characterized in that: The specific steps of step 3 are as follows: Step 3.1, select a sample in the training data set, each sample has 2000 matching point pairs, select one of the matching point pairs, each matching point pair corresponds to the left and right feature points, find the position coordinates of these two feature points on the original image in step 1.1, take the position coordinates as the center, select a range of 32 pixels in length and 32 pixels in width in the main direction of the point, segment the original image, and use the segmented image as the neighborhood image block of the feature point. Repeat the above operation to obtain the neighborhood image block of another feature point; Step 3.2, according to the result of step 3.1, scale normalization is performed on the two neighborhood image blocks of the matching point pair, and the two neighborhood image blocks are respectively input into the semantic vector generation module constructed in step 2.1 to obtain feature vectors, and the two feature vectors are normalized and added together as the semantic vector of the matching point pair; Step 3.3, repeat steps 3.1 to 3.2 to obtain the semantic vectors of all matching point pairs in the training set.

5. The feature mismatch detection method combining image semantics and coordinate relationship according to claim 4, characterized in that: The specific steps of step 4 are as follows: Step 4.1, select a sample in the training data set, select a matching point pair in the sample, obtain the position coordinates of the two feature points corresponding to the matching point pair on the original image in step 1.1, and horizontally splice and normalize the two position coordinates. Repeat the above operations to obtain the position coordinates of all matching point pairs of the sample, and vertically splice the position coordinates of all matching point pairs of the sample together to form a 2000*4 whole; Step 4.2, according to the result of step 4.1, the position coordinates of all matching point pairs of the sample are input into the coordinate position relationship vector generation module constructed in step 2.2, and the coordinate position relationship vectors of all matching point pairs can be obtained; Step 4.3, according to the results of step 4.2 and step 3.2, the coordinate position relationship vectors of all matching point pairs of the sample are normalized and added to the semantic vector to obtain the fused feature vectors of all matching point pairs of the sample; Step 4.4, repeat steps 4.1 to 4.3 to obtain the fused feature vector of all samples in the training data set.

6. The feature mismatch detection method combining image semantics and coordinate relationship according to claim 5, characterized in that: The specific steps of step 5 are as follows: Construct a parameterized neural network model, which includes three modules: weight parameter generation module, coordinate position relationship vector generation module, and matching point pair discrimination module; Step 5.1, build a weight parameter generation module with a total of 12 layers, in which convolutional layers and BN layers are used alternately, namely convolutional layer 1, BN layer 1, convolutional layer 2, BN layer 2, convolutional layer 3, BN layer 3, convolutional layer 4, BN layer 4, convolutional layer 5, BN layer 5, convolutional layer 6, BN layer 6, where the output of each layer is the input of the next layer; Step 5.2, build a coordinate position relationship vector generation module with a total of 18 layers, in which convolutional layers, IN layers, and BN layers are used alternately, namely convolutional layer 1, IN layer 1, BN layer 1, convolutional layer 2, IN layer 2, BN layer 2, convolutional layer 3, IN layer 3, BN layer 3, convolutional layer 4, IN layer 4, BN layer 4, convolutional layer 5, IN layer 5, BN layer 5, convolutional layer 6, IN layer 6, BN layer 6, where the output of each layer is the input of the next layer, and the weight parameters of convolutional layer 2 and convolutional layer 4 are the outputs of the weight parameter generation module built in step 5.1; Step 5.3, build a matching point pair discrimination module with a total of 5 layers, namely fully connected layer 1, fully connected layer 2, fully connected layer 3, fully connected layer 4, and fully connected layer 5, where the output of each layer is the input of the next layer; Step 5.4, according to the results of step 3 and step 4, a sample is selected from the training data set, and the position coordinates of all matching point pairs of the sample and the neighborhood image blocks are respectively input into the coordinate position relationship vector generation module and the weight parameter generation module in the parameterized neural network constructed in step 5 to obtain the output result of the parameterized neural network; Step 5.5, repeat the above steps 5.1-5.4 to obtain the output results of all samples.

7. The feature mismatch detection method combining image semantics and coordinate relationship according to claim 6, characterized in that: The specific steps of step 6 are as follows: Step 6.1, according to the result of step 4.4, select a sample in the training data set, input the fusion feature vector of the sample into the matching point pair discrimination module constructed in step 2.3 and perform softmax calculation on the output result. The softmax formula is shown in formula (1), where i represents the output value of the i-th node in the last layer of the neural network, e represents the exponential function, and the last layer of the neural network has two nodes, which correspond to the two situations of correct matching and incorrect matching respectively. S i The value of represents the probability that the match is a correct match or an incorrect match. The calculation result of softmax is used as the input of the cross entropy loss function, and the output value of the cross entropy loss function is calculated. This output value is the loss value. The formula of the cross entropy loss function is shown in formula (2), where p(x) is the label value, a label value of 1 represents a correct match, and a label value of 0 represents an incorrect match. q(x) is the probability that the match is a correct match or an incorrect match, and H(X, q) represents the loss value. According to the loss value, back propagation calculation is performed, and the weight bias parameters in the multimodal neural network built in step 2 are updated; H(X,q)=-∑p(x)logq(x) (2); Step 6.2, repeat step 6.1, use all the sample data in the training data set to train the multimodal neural network built in step 2 for 15 epochs, and then train the coordinate position relationship vector generation module in the network separately for 100 epochs, and adjust the learning rate parameter of the multimodal neural network according to the loss curve so that the loss curve tends to converge. After the training is completed, the weight bias parameters of the multimodal neural network are exported and saved; Step 6.3, according to the result of step 5.4, select a sample in the training data set, input the parameterized neural network output result of the sample into the matching point pair discrimination module built in step 5.3 and perform softmax calculation on the output result, use the softmax calculation result as the input of the cross entropy loss function, calculate the output value of the cross entropy loss function, and this output value is the loss value. According to the loss value, perform back propagation calculation and update the weight bias parameters in the parameterized neural network built in step 5; Step 6.4, repeat step 5.4, use all the sample data in the training data set to train the parameterized neural network built in step 5 for 15 epochs, and adjust the learning rate parameter of the parameterized neural network according to the loss curve so that the loss curve tends to converge. After the training is completed, the weight bias parameters of the parameterized neural network are exported and saved.

8. The feature mismatch detection method combining image semantics and coordinate relationship according to claim 7, characterized in that: The specific steps of step 7 are as follows: Step 7.1, select a sample to be tested, and obtain the neighborhood image block and position coordinates of the sample; Step 7.2, according to the result of step 6.2, the weight parameters are imported into the multimodal neural network constructed in step 2, the result of step 7.1 is input into the multimodal neural network, and the output result is subjected to softmax operation, and the label corresponding to the largest value is selected as the prediction result of the network; Step 7.3, according to the result of step 6.4, the weight parameters are imported into the parameterized neural network constructed in step 5, the result of step 7.1 is input into the parameterized neural network, and the output result is subjected to softmax operation, and the label corresponding to the largest value is selected as the prediction result of the network; Step 7.4, according to the result obtained in step 7.2, if the predicted label is right, then this match is a correct match; if the predicted label is wrong, then this match is an incorrect match; Step 7.5, repeat steps 7.1, 7.2, and 7.4 to obtain the prediction results of all the samples to be tested of the network built in step 2; Step 7.6, repeat steps 7.1, 7.3, and 7.4 to obtain the prediction results of all the test samples of the parameterized neural network constructed in step 5.

Images