Image key semantic feature point detection method

Abstract

The invention discloses an image key semantic feature point detection method, and belongs to the technical field of image key feature point detection. Aiming at the problem that a complex scene in a visual task is difficult to understand, the method comprises the following steps: firstly, constructing an optimized image key semantic feature detection model based on a neural network architecture; and then, through self-supervised learning, a data set for key feature detection is synthesized on an original image of the PASCAL Context data set. Then, the weight of semantic segmentation is trained on a PASCAL Context data set, and the weight of image key feature detection and descriptor generation is trained on a synthetic data set; and finally, carrying out key semantic feature point detection on the image through the optimal weight. According to the method, important semantic information is implicitly embedded into key feature description, so that feature representation is richer, the feature representation is excellent under the conditions of single texture and illumination variation, and beneficial contributions are made to implementation of intelligent visual tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of image key feature point detection, and in particular relates to an image key semantic feature point detection method. Background Art

[0002] Image key semantic feature point detection methods are crucial in computer vision, especially in target recognition, image matching and scene understanding. These feature points usually represent the edges, corners or other significant structures of objects, which can help the system better understand and analyze the image content. Traditional key feature detection methods perform well in some stable scenes, but have limited robustness in dealing with complex scenes with single textures and different lighting conditions. With the rise of deep learning, neural network-based methods can automatically learn richer feature representations and significantly improve detection accuracy. At the same time, by extracting feature points with important semantic information, the robustness and accuracy of the image key feature point detection algorithm can be improved, which can adapt to different task requirements and have stronger generalization capabilities. Summary of the invention

[0003] In order to solve the problem that complex scenes are difficult to understand in visual tasks, the present invention provides a method for detecting key semantic feature points of an image.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for detecting key semantic feature points of an image comprises the following steps:

[0006] Step 1: Build an optimized image key semantic feature detection model based on the neural network architecture;

[0007] Step 2, image preprocessing: Use the PASCAL Context dataset as the semantic segmentation dataset, perform feature point detection processing on the basis of the PASCAL Context dataset, and generate a synthetic dataset for self-supervised learning;

[0008] Step 3: Train the image key semantic feature detection model to obtain the optimal weights. The image key semantic feature detection model is divided into two independent training processes. The semantic segmentation model is trained and evaluated on the PASCAL Context dataset. The feature point detection and descriptor generation model is trained on the synthetic dataset and evaluated on the HPatches dataset. The model is trained using the Adam adaptive learning rate algorithm. In each training cycle, the model output is calculated through forward propagation, and then the network weights are updated through back propagation. After each cycle, the performance of the model on the validation set is evaluated, and the loss and accuracy indicators are recorded.

[0009] Step 4: The neural network model loads the optimal weights to detect key semantic feature points of the image.

[0010] Furthermore, the image key semantic feature detection model is an encoder-decoder structure, the encoder includes an input layer, a backbone network, and CAD_ASPP, and the decoder includes three branches: key point feature detection, descriptor generation, and semantic segmentation.

[0011] Furthermore, the step 1, based on the neural network architecture, constructs an optimized image key semantic feature detection model, including the following steps:

[0012] Step 1.1, input an image from the input layer and resize the image to H×W;

[0013] Step 1.2, then process the image through the MobileNetV3 backbone network, and output low-level feature maps with sizes of H / 4, W / 4, 24, medium-level feature maps with sizes of H / 8, W / 8, 40, and high-level feature maps with sizes of H / 16, W / 16, 80;

[0014] Step 1.3, input the high-level feature map into the densely connected convolutional layer of the CAD_ASPP module for processing;

[0015] Step 1.4, semantic segmentation branch: The output of the CAD_ASPP module is upsampled by 2 times after 1×1 convolution, and a feature map of size H / 8, W / 8, 40 is output. This feature map is spliced ​​with the intermediate feature map through the DWF module to obtain feature map a; feature map a is upsampled by 2 times after 1×1 convolution to obtain feature map c; at the same time, the intermediate feature map is upsampled by 2 times after 1×1 convolution, and spliced ​​with the low-level feature map through the DWF module to obtain feature map b; the feature map obtained by splicing feature map b and feature map c is upsampled by 4 times after 3×3 convolution to obtain the segmentation result;

[0016] Step 1.5, key feature detection branch: The intermediate feature map obtained in step 1.2 is sequentially subjected to 3×3 convolution and 1×1 convolution, and the score of each pixel of the image is output through the Softmax function, and the score is restored to the size of H, W, 1; Non-maximum suppression NMS is used to remove redundant key points, and each pixel is processed in a loop, and the scores are compared within a 9×9 range centered on the current pixel. If the current pixel has the highest score, the pixel score is retained, and the scores of the remaining pixels are modified to 0. If the current pixel does not have the highest score, the next pixel is processed according to the above steps until the loop ends; All retained pixel scores are sorted from high to low, and the first 100 pixels are taken as key feature points;

[0017] Step 1.6, generate the descriptor branch; the intermediate feature map obtained in step 1.2 is first subjected to 3×3 convolution and 1×1 convolution in sequence to obtain a feature map of size H / 8, W / 8, 256; the feature map is expanded to size H, W, 256 through bilinear interpolation upsampling; finally, the 256-dimensional descriptor of each pixel is L2 normalized to obtain the final feature descriptor.

[0018] Furthermore, the structure of the CAD_ASPP module in step 1.3 is:

[0019] The first layer is a 1×1 convolution, which outputs a feature map of size H / 16, W / 16, 112;

[0020] The second layer is a 3×3 depth-wise separable atrous convolution with a dilation rate of 3, and the input is the feature map output by the first layer;

[0021] The third layer is a 3×3 depth-wise separable atrous convolution with a dilation rate of 6, and its input is the feature map obtained by concatenating the output of the first layer and the output of the second layer;

[0022] The fourth layer is a 3×3 depth-wise separable atrous convolution with a dilation rate of 12, and its input is the concatenated feature map of the first layer output, the second layer output, and the third layer output;

[0023] The fifth layer is a 3×3 depth-wise separable atrous convolution with a dilation rate of 18, and its input is the concatenated feature map of the first layer output, the second layer output, the third layer output, and the fourth layer output;

[0024] The feature map output by the fifth layer undergoes a 1×1 convolution to obtain a feature map of size H / 16, W / 16,560; it is then processed by the CoordinateAttention mechanism to output the final result of the CAD_ASPP module.

[0025] Furthermore, the Coordinate Attention mechanism is specifically as follows: the input feature map passes the input information to the subsequent layers through the Residual mechanism, and at the same time, the input information is retained to the last layer bypassing the intermediate layer; the input feature map is average pooled in the horizontal and vertical directions to obtain two 1D vectors, and the two vectors are concatenated and resized to 1, (W + H) / 16, 35 through 1 × 1 convolution; the concatenated vectors are separately encoded into direction-aware attention and position attention through BatchNorm normalization and H-Swish activation function, and the sizes are adjusted through 1 × 1 convolution respectively; finally, the sigmoid activation function is used to transform the direction-aware attention and position attention into attention weights, and the attention weights are fused with the original input information retained by the Residual through the Re-Weight method to obtain the final output.

[0026] Further, the processing flow of the DWF module is as follows: the high-level feature map is subjected to 3×3 convolution and then up-sampled by bilinear interpolation to perform linear interpolation in the horizontal and vertical directions respectively; the up-sampled feature map and the intermediate feature map are fused by multiplication; the fused feature map is subjected to 3×3 convolution, and then the output of the convolution layer is subjected to nonlinear transformation using the ReLU activation function; the transformed fused feature is subjected to 1×1 convolution to adjust the channel and then two weight masks are generated using the softmax function, which respectively represent the importance of the high-level feature and the intermediate feature in the fused feature; the corresponding weight mask is mapped to the original feature to obtain the weighted feature, and finally the weighted feature is fused by addition; the weighted feature is subjected to 1×1 convolution to adjust the channel and then the feature map is output;

[0027] The DWF module processing of the intermediate feature map and the low-level feature map, and the high-level feature map and the low-level feature map is similar.

[0028] Furthermore, the synthetic data set generation process in step 2 is as follows:

[0029] Step 2.1, creating a binary image of a simple geometric shape, where the white background pixel value is 255 and the black foreground pixel value is 0. The simple geometric shape includes a point, a straight line, a rectangle, a square, a circle, and a triangle. The boundaries, intersections, and corners of the simple geometric shape are feature points.

[0030] Step 2.2, using the image key semantic feature detection model, training on the binary image to obtain the key feature weights;

[0031] In step 2.3, pseudo labels are generated in the PASCAL Context dataset using key feature weights to obtain a synthetic dataset.

[0032] The PASCAL Context dataset contains 10,103 images, of which 4,998 are used for training sets and 5,105 are used for validation sets. Each image has corresponding pixel-level semantic annotations.

[0033] Furthermore, in step 3, the model is trained using the Adam adaptive learning rate algorithm, with an initial learning rate of 5e-4 and a total of 100 rounds of training.

[0034] Furthermore, the loss in step 3 is calculated by the following loss function:

[0035] ① Feature point detection loss function:

[0036]

[0037] X is the predicted key feature point probability map, Y is the real label map, and their sizes are (H / 8, W / 8, 65), L p(X,Y) Calculate the difference between X and Y, N p represents the number of all pixels in X, exp(x iy ) represents the sum of the probabilities of the 65 channels of the i-th pixel in X being the true label, exp(x ij ) represents the probability of the jth channel of the i-th pixel in X, represents the sum of all probabilities of the 65 channels of the i-th pixel in X;

[0038] ②Descriptor loss function:

[0039]

[0040]

[0041] in represents the perspective transformation matrix, C g is the coordinate corresponding to the g-th pixel in the original image, C′ g′ is the coordinate corresponding to the g'th pixel in the matching image. The descriptor distance between the original image and the matching image after perspective transformation is expressed as r represents the correspondence between two descriptors, 1 represents similarity, and 0 represents dissimilarity. and are the descriptor sets on the original image and the matching image, respectively. R is the set of all corresponding relationships. The descriptor similarity between the matching pairs is calculated. N d express The number of all descriptors in , d T d′ is used to measure the similarity between two descriptors, m p Indicates the similarity boundary threshold of the matching point, which is greater than m p Indicates similarity, m n Indicates the similarity boundary threshold of non-matching points, less than m n Indicates dissimilarity;

[0042] ③Semantic segmentation loss function:

[0043]

[0044] p represents the predicted value; q represents the true value, p h and q h They represent the predicted value and true value of the h-th pixel respectively.

[0045] 10. The method for detecting key semantic feature points of an image according to claim 1, wherein the accuracy in step 3 includes the accuracy of semantic segmentation and the accuracy of feature point detection;

[0046] Semantic segmentation accuracy is evaluated by MloU and MPA.

[0047]

[0048] MloU represents the ratio of the intersection and union of the true value and predicted value of the semantic label; MPA represents the proportion of correctly classified pixels in each class; where N k represents the number of semantic categories; k represents semantic category k, k' represents other categories, E kk represents the number of pixels correctly predicted for category k, E kk' represents the number of pixels whose actual category is k but is incorrectly predicted to be other categories, E k'k Represents the number of pixels that are actually other categories but are incorrectly predicted as category k;

[0049] The accuracy of feature point detection is evaluated by Repeatability, Matching Score, Mean Localization Error, and NearestNeighbormeanAverage Precision:

[0050]

[0051] Repeatability measures the consistency of feature points under different perspectives, scales or image changes; among them, S r represents the number of feature points detected at the same position in the two images, S represents the total number of feature points detected in the original image, S' represents the total number of feature points detected in the matching image, and min(S, S') represents the minimum number of feature points detected in the two images;

[0052]

[0053] Matching Score measures the accuracy of feature point detection and descriptor matching, where S m represents the number of correctly matched feature point pairs, and S' represents the total number of feature points detected in the matching image;

[0054]

[0055] MLE measures the average position error between the predicted key feature points and the true key feature points; S represents the number of all feature points in the original image, T o is the position of the oth feature point predicted by the model, Z o is the true position of the oth feature point; ||T o -Z o || represents the Euclidean distance between the predicted key feature points and the true key feature points;

[0056]

[0057]

[0058] NN mAP is an important indicator for evaluating the matching performance of feature descriptors, and is used to measure the distinguishing ability of feature descriptors; TP is the number of samples correctly classified as positive examples, FP is the number of samples incorrectly classified as positive examples, and FN is the number of samples incorrectly classified as negative examples. AP calculates the average precision of each feature descriptor, and determines whether two feature descriptors are of the same class by setting different distance thresholds. TS is the number of different thresholds, u is the current threshold index, and S is the distance between the two feature descriptors. d is the number of descriptors in the predicted sample, and w is the current descriptor index.

[0059] Compared with the prior art, the present invention has the following advantages:

[0060] 1. The present invention implicitly embeds semantic information into feature descriptors.

[0061] 2. The present invention integrates semantic segmentation, key feature detection, and descriptor generation into one network, reducing the complexity of the model.

[0062] 3. In the CAD_ASPP module optimized by the present invention, feature fusion is optimized from the original parallel splicing to a dense connection mode of series plus parallel. Small dilation rate convolution collects dense features, large dilation rate convolution increases the receptive field, and the multi-scale receptive fields that increase layer by layer share information in a cascade manner to obtain multi-scale dense features. In order to reduce the amount of model calculation, ordinary hole convolution is optimized to depth-separable hole convolution; the pooling layer is discarded to maintain high resolution and more delicate output; CoordinateAttention is introduced to fuse the contextual information of deep features and improve image segmentation performance, capturing global spatial coordinate information while maintaining resolution.

[0063] 4. In the DWF module designed by the present invention, the semantic embedding of adjacent features is performed by pixel-by-pixel multiplication to enhance their shared semantic information; the fused features use the softmax function to dynamically allocate feature weights, and redundant information is suppressed by weighted fusion.

[0064] 5. This paper designs a new lightweight semantic segmentation model with only 2.92M parameters. Compared with the initial model with MobileNetV2 as the backbone network on the PASCALContext public dataset, the MloU is improved by 9.38% and the MPA is improved by 14.73%;

[0065] 6. Key feature point detection and generation The descriptor is trained on the PASCAL Context dataset and evaluated using the HPatches dataset. The evaluation results are as follows: Repeatability is 0.54, MLE is 1.16, Matching Score is 0.46, and nnmAP is 0.8. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is the overall structure diagram of the key semantic feature detection model;

[0067] Figure 2 This is the backbone network structure diagram;

[0068] Figure 3 This is the CAD_ASPP module structure diagram;

[0069] Figure 4 It is the CoordinateAttention structure diagram;

[0070] Figure 5 It is the DWF module structure diagram;

[0071] Figure 6 It is a branch structure diagram of semantic segmentation;

[0072] Figure 7 Detect key feature points and generate descriptor branch structure diagram;

[0073] Figure 8 This is the result diagram of key semantic feature point detection. DETAILED DESCRIPTION

[0074] In order to gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description of the present invention. However, the present invention has multiple implementations and is not limited to the specific examples listed herein. The presentation of these examples is intended to deepen the comprehensive understanding of the disclosure of the present invention.

[0075] Example 1

[0076] A method for detecting key semantic feature points of an image comprises the following steps:

[0077] Step 1: Build an optimized image key semantic feature detection model based on the neural network architecture;

[0078] The overall structure of the model is as follows Figure 1As shown in the figure, the image key semantic feature detection model is an encoder-decoder structure. The encoder includes an input layer, a backbone network, and CAD_ASPP. The decoder includes three branches: key point feature detection, descriptor generation, and semantic segmentation. The key feature detection branch generates key feature point coordinate information, and the semantic information generated by semantic segmentation is implicitly embedded in the feature descriptor to generate a key semantic feature descriptor.

[0079] Step 1.1, input an image from the input layer and resize the image to H×W;

[0080] Step 1.2, then process the image through the MobileNetV3 backbone network, and output low-level feature maps with sizes of H / 4, W / 4, 24, medium-level feature maps with sizes of H / 8, W / 8, 40, and high-level feature maps with sizes of H / 16, W / 16, 80;

[0081] Step 1.3, input the high-level feature map into the densely connected convolutional layer of the CAD_ASPP module for processing; the CAD_ASPP module structure diagram is shown in Figure 3 As shown, the structure of the CAD_ASPP module is:

[0082] The first layer is a 1×1 convolution, which outputs a feature map of size H / 16, W / 16, 112;

[0083] The second layer is a 3×3 depth-wise separable atrous convolution with a dilation rate of 3, and the input is the feature map output by the first layer;

[0084] The third layer is a 3×3 depth-wise separable atrous convolution with a dilation rate of 6, and its input is the feature map obtained by concatenating the output of the first layer and the output of the second layer;

[0085] The fourth layer is a 3×3 depth-wise separable atrous convolution with a dilation rate of 12, and its input is the concatenated feature map of the first layer output, the second layer output, and the third layer output;

[0086] The fifth layer is a 3×3 depth-wise separable atrous convolution with a dilation rate of 18, and its input is the concatenated feature map of the first layer output, the second layer output, the third layer output, and the fourth layer output;

[0087] The feature map output by the fifth layer undergoes a 1×1 convolution to obtain a feature map of size H / 16, W / 16,560; it is then processed by the CoordinateAttention mechanism to output the final result of the CAD_ASPP module.

[0088] The processing flow of the Coordinate Attention mechanism is as follows Figure 4As shown in the figure, the input feature map passes the input information to the subsequent layers through the Residual mechanism, and at the same time, the input information is retained to the last layer by bypassing the intermediate layers; the input feature map is average pooled in the horizontal and vertical directions to obtain two 1D vectors, which are concatenated and resized to (1, (W+H) / 16, 35) through 1×1 convolution; the concatenated vectors are separately encoded into direction-aware attention and position-aware attention through BatchNorm normalization and H-Swish activation function, and the sizes are adjusted through 1×1 convolution respectively; finally, the sigmoid activation function is used to transform the direction-aware attention and position-aware attention into attention weights, and the attention weights are fused with the original input information retained by the Residual through the Re-Weight method to obtain the final output;

[0089] Step 1.4, semantic segmentation branch: The semantic segmentation branch structure is as follows Figure 6 As shown in the figure, the output of the CAD_ASPP module is upsampled by 2 times after 1×1 convolution, and a feature map of size (H / 8, W / 8, 40) is output. The feature map and the intermediate feature map are spliced ​​with the DWF module to obtain feature map a; feature map a is upsampled by 2 times after 1×1 convolution to obtain feature map c; at the same time, the intermediate feature map is upsampled by 2 times after 1×1 convolution, and spliced ​​with the low-level feature map through the DWF module to obtain feature map b; the feature map obtained by splicing feature map b and feature map c is upsampled by 4 times after 3×3 convolution to obtain the segmentation result;

[0090] Taking the advanced feature map and the intermediate feature map as an example, the DWF module processing flow is as follows: Figure 5 As shown in the figure, the high-level feature map is linearly interpolated in the horizontal and vertical directions respectively by bilinear interpolation upsampling after 3×3 convolution; the upsampled feature map and the intermediate feature map are fused by multiplication; the fused feature map is convolved by 3×3, and then the output of the convolution layer is nonlinearly transformed by the ReLU activation function; the transformed fused feature is adjusted through 1×1 convolution and then two weight masks are generated by the softmax function, which respectively represent the importance of the high-level feature and the intermediate feature in the fused feature; the corresponding weight mask is mapped to the original feature to obtain the weighted feature, and finally the weighted feature is fused by addition; the weighted feature is adjusted through 1×1 convolution and then the feature map is output;

[0091] The DWF module processing of the intermediate feature map and the low-level feature map, and the high-level feature map and the low-level feature map is similar.

[0092] Step 1.5, key feature detection branch: The key feature detection branch is as follows Figure 7As shown in the figure, the intermediate feature map obtained in step 1.2 is sequentially subjected to 3×3 convolution and 1×1 convolution, and the score of each pixel of the image is output through the Softmax function, and the score is restored to the size of H, W, 1; the non-maximum suppression NMS is used to remove redundant key points, and each pixel is processed cyclically, and the scores are compared within a 9×9 range centered on the current pixel. If the current pixel has the highest score, the pixel score is retained, and the scores of the remaining pixels are modified to 0. If the current pixel does not have the highest score, the next pixel is processed according to the above steps until the loop ends; the scores of all retained pixels are sorted from high to low, and the first 100 pixels are taken as key feature points;

[0093] Step 1.6, generate a description sub-branch; generate a description sub-branch such as Figure 7 As shown in Figure 1, the intermediate feature map obtained in step 1.2 is first subjected to 3×3 convolution and 1×1 convolution in sequence to obtain a feature map of size (H / 8, W / 8, 256); the feature map is expanded to (H, W, 256) by bilinear interpolation upsampling; finally, the 256-dimensional descriptor of each pixel is L2 normalized to obtain the final feature descriptor.

[0094] Step 2, image preprocessing: Use the PASCAL Context dataset as the semantic segmentation dataset, perform feature point detection on the basis of the PASCAL Context dataset, and generate a synthetic dataset for self-supervised learning; the semantic segmentation dataset uses the PASCAL Context dataset, which contains 10,103 images, of which 4,998 are used for training sets and 5,105 are used for validation sets. Each image has corresponding pixel-level semantic annotations; the dataset for feature point detection and descriptor generation is synthesized on the original image of the PASCAL Context dataset through self-supervised learning;

[0095] The synthetic dataset generation process is as follows:

[0096] Step 2.1, creating a binary image of a simple geometric shape, where the white background pixel value is 255 and the black foreground pixel value is 0. The simple geometric shape includes a point, a straight line, a rectangle, a square, a circle, and a triangle. The boundaries, intersections, and corners of the simple geometric shape are feature points.

[0097] Step 2.2, using the image key semantic feature detection model, training on the binary image to obtain the key feature weights;

[0098] In step 2.3, pseudo labels are generated in the PASCAL Context dataset using key feature weights to obtain a synthetic dataset.

[0099] Step 3: train the image key semantic feature detection model to obtain the optimal weight;

[0100] The model was trained using the Adam adaptive learning rate algorithm with an initial learning rate of 5e-4 for 100 rounds. In each training cycle, the model output was calculated through forward propagation, and then the network weights were updated through backpropagation. After each cycle, the performance of the model on the validation set was evaluated, and the loss and accuracy indicators were recorded.

[0101] The loss is calculated by the following loss function:

[0102] ① Feature point detection loss function:

[0103]

[0104] X is the predicted key feature point probability map, Y is the real label map, and their sizes are (H / 8, W / 8, 65), L p (X,Y) Calculate the difference between X and Y, N p represents the number of all pixels in X, exp(x iy ) represents the sum of the probabilities of the 65 channels of the i-th pixel in X being the true label, exp(x ij ) represents the probability of the jth channel of the i-th pixel in X, represents the sum of all probabilities of the 65 channels of the i-th pixel in X;

[0105] ②Descriptor loss function:

[0106]

[0107]

[0108] in represents the perspective transformation matrix, C g is the coordinate corresponding to the g-th pixel in the original image, C′ g′ is the coordinate corresponding to the g'th pixel in the matching image. The descriptor distance between the original image and the matching image after perspective transformation is expressed as r represents the correspondence between two descriptors, 1 represents similarity, and 0 represents dissimilarity. and are the descriptor sets on the original image and the matching image, respectively. R is the set of all corresponding relationships. The descriptor similarity between the matching pairs is calculated. N d express The number of all descriptors in , d T d′ is used to measure the similarity between two descriptors, m p Indicates the similarity boundary threshold of the matching point, which is greater than m p Indicates similarity, m n Indicates the similarity boundary threshold of non-matching points, less than mn Indicates dissimilarity;

[0109] ③Semantic segmentation loss function:

[0110]

[0111] p represents the predicted value; q represents the true value, p h and q h They represent the predicted value and true value of the h-th pixel respectively.

[0112] 10. The method for detecting key semantic feature points of an image according to claim 1, wherein the accuracy in step 3 includes the accuracy of semantic segmentation and the accuracy of feature point detection;

[0113] Semantic segmentation accuracy is evaluated by MloU and MPA.

[0114]

[0115] MloU represents the ratio of the intersection and union of the true value and predicted value of the semantic label; MPA represents the proportion of correctly classified pixels in each class; where N k represents the number of semantic categories; k represents semantic category k, k' represents other categories, E kk represents the number of pixels correctly predicted for category k, E kk' represents the number of pixels whose actual category is k but is incorrectly predicted to be other categories, E k'k Represents the number of pixels that are actually other categories but are incorrectly predicted as category k;

[0116] The accuracy of feature point detection is evaluated by Repeatability, Matching Score, Mean Localization Error, and NearestNeighbormeanAverage Precision:

[0117]

[0118] Repeatability measures the consistency of feature points under different perspectives, scales or image changes; among them, S r represents the number of feature points detected at the same position in the two images, S represents the total number of feature points detected in the original image, S' represents the total number of feature points detected in the matching image, and min(S, S') represents the minimum number of feature points detected in the two images;

[0119]

[0120] Matching Score measures the accuracy of feature point detection and descriptor matching, where S m represents the number of correctly matched feature point pairs, and S' represents the total number of feature points detected in the matching image;

[0121]

[0122] MLE measures the average position error between the predicted key feature points and the true key feature points; S represents the number of all feature points in the original image, T o is the position of the oth feature point predicted by the model, Z o is the true position of the oth feature point; ||T o -Z o || represents the Euclidean distance between the predicted key feature points and the true key feature points;

[0123]

[0124]

[0125] NN mAP is an important indicator for evaluating the matching performance of feature descriptors, and is used to measure the distinguishing ability of feature descriptors; TP is the number of samples correctly classified as positive examples, FP is the number of samples incorrectly classified as positive examples, and FN is the number of samples incorrectly classified as negative examples. AP calculates the average precision of each feature descriptor, and determines whether two feature descriptors are of the same class by setting different distance thresholds. TS is the number of different thresholds, u is the current threshold index, and S is the distance between the two feature descriptors. d is the number of descriptors in the predicted sample, and w is the current descriptor index.

[0126] Step 4: The neural network model loads the optimal weights to detect key semantic feature points of the image, such as Figure 8 shown.

[0127] The above methods can improve the robustness and accuracy of image key feature point detection and make a beneficial contribution to the realization of intelligent vision tasks.

[0128] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For the ordinary technical personnel in the field, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A method for detecting key semantic feature points of an image, characterized in that: The following steps are involved: Step 1: Build an optimized image key semantic feature detection model based on the neural network architecture; Step 2, image preprocessing: Use the PASCAL Context dataset as the semantic segmentation dataset, perform feature point detection processing on the basis of the PASCAL Context dataset, and generate a synthetic dataset for self-supervised learning; Step 3: Train the image key semantic feature detection model to obtain the optimal weights. The image key semantic feature detection model is divided into two independent training processes. The semantic segmentation model is trained and evaluated on the PASCAL Context dataset. The feature point detection and descriptor generation model is trained on the synthetic dataset and evaluated on the HPatches dataset. The model is trained using the Adam adaptive learning rate algorithm. In each training cycle, the model output is calculated through forward propagation, and then the network weights are updated through back propagation. After each cycle, the performance of the model on the validation set is evaluated, and the loss and accuracy indicators are recorded. Step 4: The neural network model loads the optimal weights to detect key semantic feature points of the image.

2. The method for detecting key semantic feature points of an image according to claim 1, characterized in that: The image key semantic feature detection model is an encoder-decoder structure, the encoder includes an input layer, a backbone network, and CAD_ASPP, and the decoder includes three branches: key point feature detection, descriptor generation, and semantic segmentation.

3. The method for detecting key semantic feature points of an image according to claim 1, characterized in that: The step 1, based on the neural network architecture, constructs an optimized image key semantic feature detection model, including the following steps: Step 1.1, input an image from the input layer and resize the image to H×W; Step 1.2, then process the image through the MobileNetV3 backbone network, and output low-level feature maps with sizes of H / 4, W / 4, 24, medium-level feature maps with sizes of H / 8, W / 8, 40, and high-level feature maps with sizes of H / 16, W / 16, 80; Step 1.3, input the high-level feature map into the densely connected convolutional layer of the CAD_ASPP module for processing; Step 1.4, semantic segmentation branch: The output of the CAD_ASPP module is upsampled by 2 times after 1×1 convolution, and a feature map of size H / 8, W / 8, 40 is output. This feature map is spliced ​​with the intermediate feature map through the DWF module to obtain feature map a; feature map a is upsampled by 2 times after 1×1 convolution to obtain feature map c; at the same time, the intermediate feature map is upsampled by 2 times after 1×1 convolution, and spliced ​​with the low-level feature map through the DWF module to obtain feature map b; the feature map obtained by splicing feature map b and feature map c is upsampled by 4 times after 3×3 convolution to obtain the segmentation result; Step 1.5, key feature detection branch: The intermediate feature map obtained in step 1.2 is sequentially subjected to 3×3 convolution and 1×1 convolution, and the score of each pixel of the image is output through the Softmax function, and the score is restored to the size of H, W, 1; Non-maximum suppression NMS is used to remove redundant key points, and each pixel is processed in a loop, and the scores are compared within a 9×9 range centered on the current pixel. If the current pixel has the highest score, the pixel score is retained, and the scores of the remaining pixels are modified to 0. If the current pixel does not have the highest score, the next pixel is processed according to the above steps until the loop ends; All retained pixel scores are sorted from high to low, and the first 100 pixels are taken as key feature points; Step 1.6, generate the descriptor branch; the intermediate feature map obtained in step 1.2 is first subjected to 3×3 convolution and 1×1 convolution in sequence to obtain a feature map of size H / 8, W / 8, 256; the feature map is expanded to size H, W, 256 through bilinear interpolation upsampling; finally, the 256-dimensional descriptor of each pixel is L2 normalized to obtain the final feature descriptor.

4. The method for detecting key semantic feature points of an image according to claim 3, characterized in that: The structure of the CAD_ASPP module in step 1.3 is: The first layer is a 1×1 convolution, which outputs a feature map of size H / 16, W / 16, 112; The second layer is a 3×3 depth-wise separable atrous convolution with a dilation rate of 3, and the input is the feature map output by the first layer; The third layer is a 3×3 depth-wise separable atrous convolution with a dilation rate of 6, and its input is the feature map obtained by concatenating the output of the first layer and the output of the second layer; The fourth layer is a 3×3 depth-wise separable atrous convolution with a dilation rate of 12, and its input is the concatenated feature map of the first layer output, the second layer output, and the third layer output; The fifth layer is a 3×3 depth-wise separable atrous convolution with a dilation rate of 18, and its input is the concatenated feature map of the first layer output, the second layer output, the third layer output, and the fourth layer output; The feature map output by the fifth layer undergoes a 1×1 convolution to obtain a feature map of size H / 16, W / 16,560; it is then processed by the CoordinateAttention mechanism to output the final result of the CAD_ASPP module.

5. The method for detecting key semantic feature points of an image according to claim 4, characterized in that: The CoordinateAttention mechanism is as follows: the input feature map passes the input information to the subsequent layers through the Residual mechanism, and at the same time, the input information is retained to the last layer bypassing the intermediate layer; the input feature map is average pooled in the horizontal and vertical directions to obtain two 1D vectors, and the two vectors are concatenated and resized to 1, (W + H) / 16, 35 through 1 × 1 convolution; the concatenated vectors are separately encoded into direction-aware attention and position attention through BatchNorm normalization and H-Swish activation function, and the sizes are adjusted through 1 × 1 convolution respectively; finally, the sigmoid activation function is used to transform the direction-aware attention and position attention into attention weights, and the attention weights are fused with the original input information retained by the Residual through the Re-Weight method to obtain the final output.

6. The method for detecting key semantic feature points of an image according to claim 3, characterized in that: The processing flow of the DWF module is as follows: the high-level feature map is subjected to 3×3 convolution and then up-sampled by bilinear interpolation to perform linear interpolation in the horizontal and vertical directions respectively; the up-sampled feature map and the intermediate feature map are fused by multiplication; the fused feature map is subjected to 3×3 convolution, and then the output of the convolution layer is nonlinearly transformed using the ReLU activation function; the transformed fused feature is subjected to 1×1 convolution to adjust the channel and then two weight masks are generated using the softmax function, which respectively represent the importance of the high-level feature and the intermediate feature in the fused feature; the corresponding weight mask is mapped to the original feature to obtain the weighted feature, and finally the weighted feature is fused by addition; The weighted features are output as feature maps after adjusting the channels through 1×1 convolution; The DWF module processing of the intermediate feature map and the low-level feature map, and the high-level feature map and the low-level feature map is similar.

7. The method for detecting key semantic feature points of an image according to claim 5, characterized in that: The process of generating the synthetic data set in step 2 is as follows: Step 2.1, creating a binary image of a simple geometric shape, where the white background pixel value is 255 and the black foreground pixel value is 0. The simple geometric shape includes a point, a straight line, a rectangle, a square, a circle, and a triangle. The boundaries, intersections, and corners of the simple geometric shape are feature points. Step 2.2, using the image key semantic feature detection model, training on the binary image to obtain the key feature weights; In step 2.3, pseudo labels are generated in the PASCAL Context dataset using key feature weights to obtain a synthetic dataset.

8. The method for detecting key semantic feature points of an image according to claim 1, characterized in that: In step 3, the model is trained using the Adam adaptive learning rate algorithm, with an initial learning rate of 5e-4 and a total of 100 rounds of training.

9. The method for detecting key semantic feature points of an image according to claim 1, characterized in that: The loss in step 3 is calculated by the following loss function: ① Feature point detection loss function: X is the predicted key feature point probability map, Y is the real label map, and their sizes are (H / 8, W / 8, 65), L p (X,Y) Calculate the difference between X and Y, N p represents the number of all pixels in X, exp(x iy ) represents the sum of the probabilities of the 65 channels of the i-th pixel in X being the true label, exp(x ij ) represents the probability of the jth channel of the i-th pixel in X, represents the sum of all probabilities of the 65 channels of the i-th pixel in X; ②Descriptor loss function: in, represents the perspective transformation matrix, C g is the coordinate corresponding to the g-th pixel in the original image, C′ g′ is the coordinate corresponding to the g'th pixel in the matching image. The descriptor distance between the original image and the matching image after perspective transformation is expressed as r represents the correspondence between two descriptors, 1 represents similarity, and 0 represents dissimilarity. and are the descriptor sets on the original image and the matching image, respectively. R is the set of all corresponding relationships. The descriptor similarity between the matching pairs is calculated. N d express The number of all descriptors in , d T d′ is used to measure the similarity between two descriptors, m p Indicates the similarity boundary threshold of the matching point, which is greater than m p Indicates similarity, m n Indicates the similarity boundary threshold of non-matching points, less than m n Indicates dissimilarity; ③Semantic segmentation loss function: Among them, p represents the predicted value; q represents the true value, p h and q h They represent the predicted value and true value of the h-th pixel respectively.

10. The method for detecting key semantic feature points of an image according to claim 1, characterized in that: The accuracy in step 3 includes the accuracy of semantic segmentation and the accuracy of feature point detection; Semantic segmentation accuracy is evaluated by MloU and MPA. MloU represents the ratio of the intersection and union of the true value and predicted value of the semantic label; MPA represents the proportion of correctly classified pixels in each class; where N k represents the number of semantic categories; k represents semantic category k, k' represents other categories, E kk represents the number of pixels correctly predicted for category k, E kk' represents the number of pixels whose actual category is k but is incorrectly predicted to be other categories, E k'k Represents the number of pixels that are actually other categories but are incorrectly predicted as category k; The accuracy of feature point detection is evaluated by Repeatability, Matching Score, Mean Localization Error, and NearestNeighbormeanAverage Precision: Repeatability measures the consistency of feature points under different perspectives, scales or image changes; among them, S r represents the number of feature points detected at the same position in the two images, S represents the total number of feature points detected in the original image, S' represents the total number of feature points detected in the matching image, and min(S, S') represents the minimum number of feature points detected in the two images; Matching Score measures the accuracy of feature point detection and descriptor matching, where S m represents the number of correctly matched feature point pairs, and S' represents the total number of feature points detected in the matching image; MLE measures the average position error between the predicted key feature points and the true key feature points; S represents the number of all feature points in the original image, T o is the position of the oth feature point predicted by the model, Z o is the true position of the oth feature point; ||T o -Z o || represents the Euclidean distance between the predicted key feature points and the true key feature points; NN mAP is an important indicator for evaluating the matching performance of feature descriptors, and is used to measure the distinguishing ability of feature descriptors; TP is the number of samples correctly classified as positive examples, FP is the number of samples incorrectly classified as positive examples, and FN is the number of samples incorrectly classified as negative examples. AP calculates the average precision of each feature descriptor, and determines whether two feature descriptors are of the same class by setting different distance thresholds. TS is the number of different thresholds, u is the current threshold index, and S is the distance between the two feature descriptors. d is the number of descriptors in the predicted sample, and w is the current descriptor index.

Images