Image key semantic feature point detection method

Abstract

The application discloses a kind of image key semantic feature point detection methods, belong to image key feature point detection technical field.Aiming at the problem that complex scene is difficult to understand in visual task, the application first constructs optimized image key semantic feature detection model based on neural network architecture;Then through self-supervised learning, the original drawing of PASCAL Context dataset is synthesized on the dataset of key feature detection.Then, the weight of semantic segmentation is trained on PASCAL Context dataset, and the weight of image key feature detection and generated descriptor is trained on the synthetic dataset.Finally, the key semantic feature point of image is detected by optimal weight.The application implicitly embeds important semantic information in key feature description, making the feature representation more rich, and performs excellently under the condition of single texture and illumination change, which makes a beneficial contribution to the realization of intelligent visual task.

Description

Technical Field

[0001] The present invention belongs to the technical field of image key feature point detection, and in particular relates to a method for detecting image key semantic feature points. Background Art

[0002] Methods for detecting key semantic feature points in images are crucial in computer vision, especially in target recognition, image matching, and scene understanding. These feature points typically represent edges, corners, or other significant structures of objects, helping the system better understand and analyze image content. Traditional key feature detection methods perform well in some stable scenes, but their robustness is limited when dealing with complex scenes with single textures and varying lighting conditions. With the rise of deep learning, neural network-based methods can automatically learn richer feature representations and significantly improve detection accuracy. At the same time, by extracting feature points with important semantic information, the robustness and accuracy of image key feature point detection algorithms can be improved, making them adaptable to different task requirements and possessing stronger generalization capabilities. Summary of the Invention

[0003] To address the problem that complex scenes are difficult to understand in visual tasks, the present invention provides a method for detecting key semantic feature points in images.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for detecting key semantic feature points of an image comprises the following steps:

[0006] Step 1: Build an optimized image key semantic feature detection model based on the neural network architecture;

[0007] Step 2, image preprocessing: Use the PASCAL Context dataset as the semantic segmentation dataset, perform feature point detection on the PASCAL Context dataset, and generate a synthetic dataset for self-supervised learning;

[0008] Step 3: Train the image key semantic feature detection model to obtain the optimal weights. The image key semantic feature detection model is divided into two independent training processes. The semantic segmentation model is trained and evaluated on the PASCAL Context dataset; the feature point detection and descriptor generation model is trained on the synthetic dataset and evaluated on the HPatches dataset. The model is trained using the Adam adaptive learning rate algorithm. In each training cycle, the model output is calculated through forward propagation, and the network weights are then updated through backpropagation. After each cycle, the model performance is evaluated on the validation set, and the loss and accuracy metrics are recorded.

[0009] Step 4: The neural network model loads the optimal weights to detect key semantic feature points in the image.

[0010] Furthermore, the image key semantic feature detection model is an encoder-decoder structure, the encoder includes an input layer, a backbone network, and CAD_ASPP, and the decoder includes three branches: key point feature detection, descriptor generation, and semantic segmentation.

[0011] Furthermore, the step 1, based on the neural network architecture, constructs an optimized image key semantic feature detection model, including the following steps:

[0012] Step 1.1: Input an image from the input layer and resize the image to H×W.

[0013] Step 1.2: The image is then processed through the MobileNetV3 backbone network, and the output sizes are low-level feature maps of H / 4, W / 4, 24, medium-level feature maps of H / 8, W / 8, 40, and high-level feature maps of H / 16, W / 16, 80.

[0014] In step 1.3, the high-level feature map is fed into the densely connected convolutional layer of the CAD_ASPP module for processing.

[0015] Step 1.4, semantic segmentation branch: The output of the CAD_ASPP module undergoes 1×1 convolution and then is upsampled by a factor of 2, outputting a feature map of size H / 8, W / 8, 40. This feature map is concatenated with the intermediate feature map through the DWF module to obtain feature map a. Feature map a undergoes 1×1 convolution and then is upsampled by a factor of 2 to obtain feature map c. At the same time, the intermediate feature map undergoes 1×1 convolution and then is upsampled by a factor of 2, and concatenated with the low-level feature map through the DWF module to obtain feature map b. The feature map obtained by concatenating feature map b and feature map c undergoes 3×3 convolution and then is upsampled by a factor of 4 to obtain the segmentation result.

[0016] Step 1.5, key feature detection branch: The intermediate feature map obtained in step 1.2 is sequentially subjected to 3×3 convolution and 1×1 convolution, and the score of each pixel in the image is output through the Softmax function, and the score is restored to the size of H, W, 1; non-maximum suppression (NMS) is used to remove redundant key points, and each pixel is processed in a loop. The scores are compared within a 9×9 range centered on the current pixel. If the current pixel has the highest score, the pixel score is retained, and the scores of the remaining pixels are modified to 0. If the current pixel does not have the highest score, the next pixel is processed according to the above steps until the end of the loop; all retained pixel scores are sorted from high to low, and the first 100 pixels are taken as key feature points;

[0017] In step 1.6, the descriptor branch is generated. The intermediate feature map obtained in step 1.2 is first subjected to 3×3 convolution and 1×1 convolution to obtain a feature map of size H / 8, W / 8, 256. The feature map is upsampled by bilinear interpolation to H, W, 256. Finally, the 256-dimensional descriptor of each pixel is L2 normalized to obtain the final feature descriptor.

[0018] Furthermore, the structure of the CAD_ASPP module in step 1.3 is:

[0019] The first layer is a 1×1 convolution, which outputs a feature map of size H / 16, W / 16, 112;

[0020] The second layer is a 3×3 depth-wise separable dilated convolution with a dilation rate of 3, and the input is the feature map output by the first layer;

[0021] The third layer is a 3×3 depth-wise separable dilated convolution with a dilation rate of 6. The input is the feature map obtained by concatenating the output of the first layer and the output of the second layer.

[0022] The fourth layer is a 3×3 depth-wise separable dilated convolution with a dilation rate of 12, and its input is the concatenated feature map of the first layer output, the second layer output, and the third layer output;

[0023] The fifth layer is a 3×3 depth-wise separable dilated convolution with a dilation rate of 18. The input is the concatenated feature map of the first layer output, the second layer output, the third layer output, and the fourth layer output.

[0024] The feature map output by the fifth layer undergoes a 1×1 convolution to obtain a feature map of size H / 16, W / 16,560; it is then processed by the CoordinateAttention mechanism to output the final result of the CAD_ASPP module.

[0025] Furthermore, the Coordinate Attention mechanism is specifically as follows: the input feature map passes the input information to the subsequent layers through the Residual mechanism, and at the same time, the input information is retained to the last layer bypassing the intermediate layers; the input feature map is average pooled in the horizontal and vertical directions to obtain two 1D vectors, which are spliced ​​and resized to 1, (W+H) / 16, 35 through 1×1 convolution; the spliced ​​vectors are separately encoded into direction-aware attention and position-aware attention through BatchNorm normalization and H-Swish activation function, and the sizes are adjusted through 1×1 convolution respectively; finally, the sigmoid activation function is used to transform the direction-aware attention and position-aware attention into attention weights, and the attention weights are fused with the original input information retained by the Residual through Re-Weight to obtain the final output.

[0026] Furthermore, the DWF module processing flow is as follows: the high-level feature map is subjected to 3×3 convolution and then up-sampled by bilinear interpolation, and then linear interpolation is performed in the horizontal and vertical directions respectively; the up-sampled feature map and the intermediate feature map are fused by multiplication; the fused feature map is subjected to 3×3 convolution, and then the output of the convolution layer is nonlinearly transformed using the ReLU activation function; the transformed fused feature is adjusted through 1×1 convolution and then two weight masks are generated using the softmax function, respectively indicating the importance of the high-level feature and the intermediate feature in the fused feature; the corresponding weight mask is mapped to the original feature to obtain the weighted feature, and finally the weighted feature is fused by addition; the weighted feature is adjusted through 1×1 convolution and then the feature map is output;

[0027] The DWF module processing of intermediate feature maps and low-level feature maps, and high-level feature maps and low-level feature maps is similar.

[0028] Furthermore, the process of generating the synthetic dataset in step 2 is as follows:

[0029] Step 2.1, create a binary image of simple geometric shapes, with a white background pixel value of 255 and a black foreground pixel value of 0. The simple geometric shapes include points, lines, rectangles, squares, circles, and triangles. The boundaries, intersections, and corners of the simple geometric shapes are feature points.

[0030] Step 2.2: Use the image key semantic feature detection model to train on binary images to obtain key feature weights;

[0031] In step 2.3, pseudo labels are generated on the PASCAL Context dataset using key feature weights to obtain a synthetic dataset.

[0032] The PASCAL Context dataset contains 10,103 images, of which 4,998 are used for training and 5,105 are used for validation. Each image has corresponding pixel-level semantic annotations.

[0033] Furthermore, in step 3, the model is trained using the Adam adaptive learning rate algorithm with an initial learning rate of 5e-4 and a total of 100 rounds of training.

[0034] Furthermore, the loss in step 3 is calculated by the following loss function:

[0035] ① Feature point detection loss function:

[0036]

[0037] X is the predicted key feature point probability map, Y is the real label map, and their sizes are (H / 8, W / 8, 65), L p(X,Y) calculates the difference between X and Y, N p Represents the number of all pixels in X, exp(x iy ) represents the sum of the probabilities of the 65 channels of the i-th pixel in X being the true label, exp(x ij ) represents the probability of the jth channel of the i-th pixel in X, represents the sum of all probabilities of the 65 channels of the i-th pixel in X;

[0038] ②Descriptor loss function:

[0039]

[0040]

[0041] in represents the perspective transformation matrix, C g is the coordinate corresponding to the g-th pixel in the original image, C′ g′ is the coordinate corresponding to the g'th pixel in the matching image. The descriptor distance between the original image after perspective transformation and the matching image is expressed as r represents the correspondence between two descriptors, 1 represents similarity, and 0 represents dissimilarity. and are the descriptor sets on the original image and the matching image, R is the set of all corresponding relationships, and the descriptor similarity between the matching pairs is calculated. d express The number of all descriptors in d T d′ is used to measure the similarity between two descriptors, m p Indicates the similarity boundary threshold of the matching point, which is greater than m p Indicates similarity, m n Indicates the similarity boundary threshold of non-matching points, which is less than m n Indicates dissimilarity;

[0042] ③Semantic segmentation loss function:

[0043]

[0044] p represents the predicted value; q represents the true value, p h and q h They represent the predicted value and true value of the h-th pixel respectively.

[0045] 10. The method for detecting key semantic feature points of an image according to claim 1, wherein the accuracy in step 3 includes the accuracy of semantic segmentation and the accuracy of feature point detection;

[0046] Semantic segmentation accuracy is evaluated by MloU and MPA.

[0047]

[0048] MloU represents the ratio of intersection and union of two sets of semantic label true value and predicted value; MPA represents the proportion of the number of correctly classified pixels in each class; wherein, N k represents the number of semantic categories; k represents semantic category k, k' represents other categories, E kk represents the number of pixels of category k that are correctly predicted, E kk' represents the number of pixels that are actually of category k but are incorrectly predicted as other categories, E k'k represents the number of pixels that are actually of other categories but are incorrectly predicted as category k;

[0049] The accuracy of feature point detection is evaluated by Repeatability, Matching Score, Mean Localization Error, Nearest Neighbor mean Average Precision:

[0050]

[0051] Repeatability measures the consistency of feature points under different perspectives, scales or image changes; wherein, S r represents the number of feature points detected at the same position in two images, S represents the total number of feature points detected in the original image, S' represents the total number of feature points detected in the matching image, and min(S, S') represents the minimum number of feature points detected in the two images;

[0052]

[0053] Matching Score measures the accuracy of feature point detection and descriptor matching, wherein, S m represents the number of correctly matched feature point pairs, and S' represents the total number of feature points detected in the matching image;

[0054]

[0055] MLE measures the average position error between the predicted key feature points and the true key feature points; S represents the number of all feature points in the original image, T o is the position of the oth feature point predicted by the model, Z o is the position of the oth true feature point; ||T o -Z o || represents the Euclidean distance between the predicted key feature points and the true key feature points;

[0056]

[0057]

[0058] NN mAP is an important indicator for evaluating the matching performance of feature descriptors and is used to measure the distinguishing ability of feature descriptors. TP is the number of samples correctly classified as positive examples, FP is the number of samples incorrectly classified as positive examples, and FN is the number of samples incorrectly classified as negative examples. AP calculates the average precision of each feature descriptor and determines whether two feature descriptors belong to the same category by setting different distance thresholds. TS is the number of different thresholds, u is the current threshold index, and S is the distance between the two descriptors. d is the number of descriptors in the predicted sample, and w is the current descriptor index.

[0059] Compared with the prior art, the present invention has the following advantages:

[0060] 1. The present invention implicitly embeds semantic information into feature descriptors.

[0061] 2. The present invention integrates semantic segmentation, key feature detection, and descriptor generation into one network, reducing the complexity of the model.

[0062] 3. In the CAD_ASPP module optimized by the present invention, feature fusion is optimized from the original parallel splicing to a dense connection mode of series plus parallel. Small dilation rate convolution collects dense features, large dilation rate convolution increases the receptive field, and the multi-scale receptive fields that increase layer by layer share information in a cascade manner to obtain multi-scale dense features. In order to reduce the amount of model calculation, ordinary void convolution is optimized to depthwise separable void convolution; the pooling layer is discarded to maintain high resolution and more delicate output; CoordinateAttention is introduced to fuse the contextual information of deep features and improve image segmentation performance, capturing global spatial coordinate information while maintaining resolution.

[0063] 4. In the DWF module designed by the present invention, semantic embedding of adjacent features is performed by pixel-by-pixel multiplication to enhance their shared semantic information; the fused features dynamically assign feature weights using the softmax function, and redundant information is suppressed through weighted fusion.

[0064] 5. This paper designs a novel lightweight semantic segmentation model with only 2.92M parameters. Compared with the initial model with MobileNetV2 as the backbone network on the PASCALContext public dataset, the MloU is improved by 9.38% and the MPA is improved by 14.73%;

[0065] 6. Key feature point detection and generation descriptors are trained on the PASCAL Context dataset and evaluated using the HPatches dataset. The evaluation results are as follows: Repeatability is 0.54, MLE is 1.16, Matching Score is 0.46, and nnmAP is 0.8. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is the overall structure diagram of the key semantic feature detection model;

[0067] Figure 2 This is the backbone network structure diagram;

[0068] Figure 3 This is the CAD_ASPP module structure diagram;

[0069] Figure 4 It is the CoordinateAttention structure diagram;

[0070] Figure 5 It is the DWF module structure diagram;

[0071] Figure 6 This is the semantic segmentation branch structure diagram;

[0072] Figure 7 Detect key feature points and generate descriptor branch structure diagram;

[0073] Figure 8 This is the result diagram of key semantic feature point detection. DETAILED DESCRIPTION

[0074] To gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description thereof. However, the present invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a comprehensive understanding of the present disclosure.

[0075] Example 1

[0076] A method for detecting key semantic feature points of an image comprises the following steps:

[0077] Step 1: Build an optimized image key semantic feature detection model based on the neural network architecture;

[0078] The overall structure of the model is as follows Figure 1As shown in Figure 2, the image key semantic feature detection model has an encoder-decoder structure. The encoder consists of an input layer, a backbone network, and CAD_ASPP. The decoder consists of three branches: key point feature detection, descriptor generation, and semantic segmentation. The key feature detection branch generates key feature point coordinate information. The semantic information generated by semantic segmentation is implicitly embedded in the feature descriptor to generate the key semantic feature descriptor.

[0079] Step 1.1: Input an image from the input layer and resize the image to H×W.

[0080] Step 1.2: The image is then processed through the MobileNetV3 backbone network, and the output sizes are low-level feature maps of H / 4, W / 4, 24, medium-level feature maps of H / 8, W / 8, 40, and high-level feature maps of H / 16, W / 16, 80.

[0081] Step 1.3: Input the high-level feature map into the densely connected convolutional layer of the CAD_ASPP module for processing; the CAD_ASPP module structure is shown in the figure below. Figure 3 As shown, the structure of the CAD_ASPP module is:

[0082] The first layer is a 1×1 convolution, which outputs a feature map of size H / 16, W / 16, 112;

[0083] The second layer is a 3×3 depth-wise separable dilated convolution with a dilation rate of 3, and the input is the feature map output by the first layer;

[0084] The third layer is a 3×3 depth-wise separable dilated convolution with a dilation rate of 6. The input is the feature map obtained by concatenating the output of the first layer and the output of the second layer.

[0085] The fourth layer is a 3×3 depth-wise separable dilated convolution with a dilation rate of 12, and its input is the concatenated feature map of the first layer output, the second layer output, and the third layer output;

[0086] The fifth layer is a 3×3 depth-wise separable dilated convolution with a dilation rate of 18. The input is the concatenated feature map of the first layer output, the second layer output, the third layer output, and the fourth layer output.

[0087] The feature map output by the fifth layer undergoes a 1×1 convolution to obtain a feature map of size H / 16, W / 16,560; it is then processed by the CoordinateAttention mechanism to output the final result of the CAD_ASPP module.

[0088] The processing flow of the Coordinate Attention mechanism is as follows Figure 4As shown in the figure, the input feature map passes the input information to the subsequent layers through the Residual mechanism, and at the same time, the input information is retained in the last layer by bypassing the intermediate layers; the input feature map is average pooled in the horizontal and vertical directions to obtain two 1D vectors, which are concatenated and resized to (1, (W+H) / 16, 35) through 1×1 convolution; the concatenated vectors are separately encoded into direction-aware attention and position-aware attention through BatchNorm normalization and H-Swish activation function, and the sizes are adjusted through 1×1 convolution respectively; finally, the direction-aware attention and position-aware attention are transformed into attention weights using the sigmoid activation function, and the attention weights are fused with the original input information retained by the Residual through the Re-Weight method to obtain the final output;

[0089] Step 1.4, semantic segmentation branch: The semantic segmentation branch structure is as follows Figure 6 As shown in the figure, the output of the CAD_ASPP module is upsampled by 2 times after 1×1 convolution, and a feature map of size (H / 8, W / 8, 40) is output. This feature map is spliced ​​with the intermediate feature map through the DWF module to obtain feature map a; feature map a is upsampled by 2 times after 1×1 convolution to obtain feature map c; at the same time, the intermediate feature map is upsampled by 2 times after 1×1 convolution, and spliced ​​with the low-level feature map through the DWF module to obtain feature map b; the feature map obtained by splicing feature map b and feature map c is upsampled by 4 times after 3×3 convolution to obtain the segmentation result;

[0090] Taking the advanced feature map and the intermediate feature map as an example, the DWF module processing flow is as follows: Figure 5 As shown in the figure, the high-level feature map is linearly interpolated in the horizontal and vertical directions respectively by bilinear interpolation upsampling after 3×3 convolution; the upsampled feature map and the intermediate feature map are fused by multiplication; the fused feature map is convolved by 3×3, and then the output of the convolution layer is nonlinearly transformed using the ReLU activation function; the transformed fused feature is adjusted through 1×1 convolution and two weight masks are generated using the softmax function, which respectively represent the importance of the high-level feature and the intermediate feature in the fused feature; the corresponding weight mask is mapped to the original feature to obtain the weighted feature, and finally the weighted feature is fused by addition; the weighted feature is adjusted through 1×1 convolution and the channel is output as a feature map;

[0091] The DWF module processing of intermediate feature maps and low-level feature maps, and high-level feature maps and low-level feature maps is similar.

[0092] Step 1.5, key feature detection branch: The key feature detection branch is as follows Figure 7As shown in the figure, the intermediate feature map obtained in step 1.2 is sequentially subjected to 3×3 convolution and 1×1 convolution, and the score of each pixel of the image is output through the Softmax function, and the score is restored to the size of H, W, 1; non-maximum suppression NMS is used to remove redundant key points, and each pixel is processed cyclically. The scores are compared within a 9×9 range centered on the current pixel. If the current pixel has the highest score, the pixel score is retained, and the scores of the remaining pixels are modified to 0. If the current pixel does not have the highest score, the next pixel is processed according to the above steps until the end of the loop; all retained pixel scores are sorted from high to low, and the first 100 pixels are taken as key feature points;

[0093] Step 1.6, generate a descriptor branch; generate a descriptor branch such as Figure 7 As shown in Figure 1, the intermediate feature map obtained in step 1.2 is first subjected to 3×3 convolution and 1×1 convolution to obtain a feature map of size (H / 8, W / 8, 256); the feature map is expanded to (H, W, 256) by bilinear interpolation upsampling; finally, the 256-dimensional descriptor of each pixel is L2 normalized to obtain the final feature descriptor.

[0094] Step 2: Image preprocessing: The PASCAL Context dataset is used as the semantic segmentation dataset. Feature point detection is performed on the PASCAL Context dataset to generate a synthetic dataset for self-supervised learning. The semantic segmentation dataset uses the PASCAL Context dataset, which contains 10,103 images, of which 4,998 are used for training and 5,105 are used for validation. Each image has corresponding pixel-level semantic annotations. The dataset for feature point detection and generated descriptors is synthesized on the original image of the PASCAL Context dataset through self-supervised learning.

[0095] The synthetic dataset generation process is as follows:

[0096] Step 2.1, create a binary image of simple geometric shapes, with a white background pixel value of 255 and a black foreground pixel value of 0. The simple geometric shapes include points, lines, rectangles, squares, circles, and triangles. The boundaries, intersections, and corners of the simple geometric shapes are feature points.

[0097] Step 2.2: Use the image key semantic feature detection model to train on binary images to obtain key feature weights;

[0098] In step 2.3, pseudo labels are generated on the PASCAL Context dataset using key feature weights to obtain a synthetic dataset.

[0099] Step 3: Train the image key semantic feature detection model to obtain the optimal weight;

[0100] The model was trained using the Adam adaptive learning rate algorithm with an initial learning rate of 5e-4 for 100 epochs. In each training cycle, the model output was calculated through forward propagation, and the network weights were then updated through backpropagation. After each epoch, the model performance was evaluated on the validation set, and the loss and accuracy metrics were recorded.

[0101] The loss is calculated by the following loss function:

[0102] ① Feature point detection loss function:

[0103]

[0104] X is the predicted key feature point probability map, Y is the real label map, and their sizes are (H / 8, W / 8, 65), L p (X,Y) calculates the difference between X and Y, N p Represents the number of all pixels in X, exp(x iy ) represents the sum of the probabilities of the 65 channels of the i-th pixel in X being the true label, exp(x ij ) represents the probability of the jth channel of the i-th pixel in X, represents the sum of all probabilities of the 65 channels of the i-th pixel in X;

[0105] ②Descriptor loss function:

[0106]

[0107]

[0108] in represents the perspective transformation matrix, C g is the coordinate corresponding to the g-th pixel in the original image, C′ g′ is the coordinate corresponding to the g'th pixel in the matching image. The descriptor distance between the original image after perspective transformation and the matching image is expressed as r represents the correspondence between two descriptors, 1 represents similarity, and 0 represents dissimilarity. and are the descriptor sets on the original image and the matching image, R is the set of all corresponding relationships, and the descriptor similarity between the matching pairs is calculated. d express The number of all descriptors in d T d′ is used to measure the similarity between two descriptors, m p Indicates the similarity boundary threshold of the matching point, which is greater than m p Indicates similarity, m n Indicates the similarity boundary threshold of non-matching points, which is less than mn Indicates dissimilarity;

[0109] ③Semantic segmentation loss function:

[0110]

[0111] p represents the predicted value; q represents the true value, p h and q h They represent the predicted value and true value of the h-th pixel respectively.

[0112] 10. The method for detecting key semantic feature points of an image according to claim 1, wherein the accuracy in step 3 includes the accuracy of semantic segmentation and the accuracy of feature point detection;

[0113] Semantic segmentation accuracy is evaluated by MloU and MPA.

[0114]

[0115] MloU represents the ratio of the intersection and union of the semantic label true value and predicted value; MPA represents the proportion of correctly classified pixels in each class; where N k represents the number of semantic categories; k represents semantic category k, k' represents other categories, E kk represents the number of pixels correctly predicted for category k, E kk' represents the number of pixels whose actual category is k but is incorrectly predicted to be other categories, E k'k represents the number of pixels that are actually other categories but are incorrectly predicted to be category k;

[0116] The accuracy of feature point detection is evaluated by Repeatability, Matching Score, Mean Localization Error, and NearestNeighbormeanAveragePrecision:

[0117]

[0118] Repeatability measures the consistency of feature points under different perspectives, scales or image changes; among them, S r Indicates the number of feature points detected at the same position in the two images, S represents the total number of feature points detected in the original image, S' represents the total number of feature points detected in the matching image, and min(S, S') represents the minimum number of feature points detected in the two images;

[0119]

[0120] Matching Score measures the accuracy of feature point detection and descriptor matching, where S m represents the number of correctly matched feature point pairs, and S' represents the total number of feature points detected in the matching image;

[0121]

[0122] MLE measures the average position error between the predicted key feature points and the true key feature points; S represents the number of all feature points in the original image, T o is the position of the oth feature point predicted by the model, Z o is the true position of the oth feature point; ||T o -Z o || represents the Euclidean distance between the predicted key feature points and the true key feature points;

[0123]

[0124]

[0125] NN mAP is an important indicator for evaluating the matching performance of feature descriptors and is used to measure the distinguishing ability of feature descriptors. TP is the number of samples correctly classified as positive examples, FP is the number of samples incorrectly classified as positive examples, and FN is the number of samples incorrectly classified as negative examples. AP calculates the average precision of each feature descriptor and determines whether two feature descriptors belong to the same category by setting different distance thresholds. TS is the number of different thresholds, u is the current threshold index, and S is the distance between the two descriptors. d is the number of descriptors in the predicted sample, and w is the current descriptor index.

[0126] Step 4: The neural network model loads the optimal weights to detect key semantic feature points of the image, such as Figure 8 shown.

[0127] The above methods can improve the robustness and accuracy of image key feature point detection and make a beneficial contribution to the realization of intelligent vision tasks.

[0128] Any matters not described in detail in this specification are prior art known to those skilled in the art. Although the above description of the present invention is based on specific embodiments to facilitate understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the specific embodiments. As long as various modifications are within the spirit and scope of the present invention as defined and determined by the appended claims, such modifications will be obvious to those skilled in the art, and all inventions and creations utilizing the concepts of the present invention are protected.

Claims

1. A method for detecting key semantic feature points of an image, characterized in that: The following steps are involved: Step 1: Build an optimized image key semantic feature detection model based on the neural network architecture; specifically, the following steps are included: Step 1.1: The image is then processed through the MobileNetV3 backbone network, and the output sizes are low-level feature maps of H / 4, W / 4, 24, medium-level feature maps of H / 8, W / 8, 40, and high-level feature maps of H / 16, W / 16, 80. In step 1.2, the high-level feature map is fed into the densely connected convolutional layer of the CAD_ASPP module for processing. Step 1.3, semantic segmentation branch: The output of the CAD_ASPP module undergoes 1×1 convolution and then is upsampled by a factor of 2, outputting a feature map of size H / 8, W / 8, 40. This feature map is concatenated with the intermediate feature map through the DWF module to obtain feature map a. Feature map a undergoes 1×1 convolution and then is upsampled by a factor of 2 to obtain feature map c. At the same time, the intermediate feature map undergoes 1×1 convolution and then is upsampled by a factor of 2, and concatenated with the low-level feature map through the DWF module to obtain feature map b. The feature map obtained by concatenating feature map b and feature map c undergoes 3×3 convolution and then is upsampled by a factor of 4 to obtain the segmentation result. Step 2, image preprocessing: Use the PASCAL Context dataset as the semantic segmentation dataset, perform feature point detection on the PASCAL Context dataset, and generate a synthetic dataset for self-supervised learning; Step 3: Train the image key semantic feature detection model to obtain the optimal weights. The image key semantic feature detection model is divided into two independent training processes. The semantic segmentation model is trained and evaluated on the PASCAL Context dataset; the feature point detection and descriptor generation model is trained on the synthetic dataset and evaluated on the HPatches dataset. The model is trained using the Adam adaptive learning rate algorithm. In each training cycle, the model output is calculated through forward propagation, and the network weights are then updated through backpropagation. After each cycle, the model performance is evaluated on the validation set, and the loss and accuracy metrics are recorded. Step 4: The neural network model loads the optimal weights to detect key semantic feature points of the image; The DWF module processing flow is as follows: the high-level feature map is subjected to 3×3 convolution and then upsampled using bilinear interpolation to perform linear interpolation in the horizontal and vertical directions respectively; the upsampled feature map is fused with the intermediate feature map by multiplication; the fused feature map is subjected to 3×3 convolution, and then the output of the convolution layer is nonlinearly transformed using the ReLU activation function; the transformed fused feature is adjusted through 1×1 convolution and then two weight masks are generated using the softmax function to represent the importance of the high-level feature and the intermediate feature in the fused feature respectively; the corresponding weight mask is mapped to the original feature to obtain the weighted feature, and finally the weighted feature is fused by addition; the weighted feature is adjusted through 1×1 convolution and the feature map is output; The DWF module processing of intermediate feature maps and low-level feature maps, and high-level feature maps and low-level feature maps is similar.

2. The method for detecting key semantic feature points in an image according to claim 1, wherein: The image key semantic feature detection model is an encoder-decoder structure, the encoder includes an input layer, a backbone network, and CAD_ASPP, and the decoder includes three branches: key point feature detection, descriptor generation, and semantic segmentation.

3. The method for detecting key semantic feature points of an image according to claim 1, wherein: The step 1, constructing an optimized image key semantic feature detection model based on a neural network architecture, further includes the following steps: Step 1.4, key feature detection branch: The intermediate feature map obtained in step 1.1 is sequentially subjected to 3×3 convolution and 1×1 convolution, and the score of each pixel in the image is output through the Softmax function, and the score is restored to the size of H, W, 1; non-maximum suppression (NMS) is used to remove redundant key points, and each pixel is processed in a loop. The scores are compared within a 9×9 range centered on the current pixel. If the current pixel has the highest score, the pixel score is retained, and the scores of the remaining pixels are modified to 0. If the current pixel does not have the highest score, the next pixel is processed according to the above steps until the end of the loop; all retained pixel scores are sorted from high to low, and the first 100 pixels are taken as key feature points; In step 1.5, a descriptor branch is generated. The intermediate feature map obtained in step 1.1 is first subjected to 3×3 convolution and 1×1 convolution to obtain a feature map of size H / 8, W / 8, 256. This feature map is upsampled to H, W, 256 by bilinear interpolation. Finally, the 256-dimensional descriptor of each pixel is L2 normalized to obtain the final feature descriptor.

4. The method for detecting key semantic feature points of an image according to claim 1, wherein: The structure of the CAD_ASPP module is: The first layer is a 1×1 convolution, which outputs a feature map of size H / 16, W / 16, 112; The second layer is a 3×3 depth-wise separable dilated convolution with a dilation rate of 3, and the input is the feature map output by the first layer; The third layer is a 3×3 depth-wise separable dilated convolution with a dilation rate of 6. The input is the feature map obtained by concatenating the output of the first layer and the output of the second layer. The fourth layer is a 3×3 depth-wise separable dilated convolution with a dilation rate of 12, and its input is the concatenated feature map of the first layer output, the second layer output, and the third layer output; The fifth layer is a 3×3 depth-wise separable dilated convolution with a dilation rate of 18. The input is the concatenated feature map of the first layer output, the second layer output, the third layer output, and the fourth layer output. The feature map output by the fifth layer undergoes a 1×1 convolution to obtain a feature map of size H / 16, W / 16,560; it is then processed by the Coordinate Attention mechanism to output the final result of the CAD_ASPP module.

5. The method for detecting key semantic feature points of an image according to claim 4, wherein: The Coordinate Attention mechanism is specifically as follows: the input feature map passes the input information to the subsequent layers through the Residual mechanism, and at the same time retains the input information to the last layer by bypassing the intermediate layers; the input feature map is average pooled in the horizontal and vertical directions to obtain two 1D vectors, which are spliced ​​and resized to 1, (W+H) / 16, 35 through 1×1 convolution; the spliced ​​vectors are separately encoded into direction-aware attention and position-aware attention through BatchNorm normalization and H-Swish activation function, and the sizes are adjusted through 1×1 convolution respectively; finally, the direction-aware attention and position-aware attention are transformed into attention weights using the sigmoid activation function, and the attention weights are fused with the original input information retained by the Residual through the Re-Weight method to obtain the final output.

6. The method for detecting key semantic feature points of an image according to claim 5, wherein: The process of generating the synthetic data set in step 2 is as follows: Step 2.1, create a binary image of simple geometric shapes, with a white background pixel value of 255 and a black foreground pixel value of 0. The simple geometric shapes include points, lines, rectangles, squares, circles, and triangles. The boundaries, intersections, and corners of the simple geometric shapes are feature points. Step 2.2: Use the image key semantic feature detection model to train on binary images to obtain key feature weights; In step 2.3, pseudo labels are generated on the PASCAL Context dataset using key feature weights to obtain a synthetic dataset.

7. The method for detecting key semantic feature points of an image according to claim 1, wherein: In step 3, the model is trained using the Adam adaptive learning rate algorithm with an initial learning rate of 5e-4 and a total of 100 rounds of training.

8. The method for detecting key semantic feature points of an image according to claim 1, wherein: The loss in step 3 is calculated using the following loss function: ① Feature point detection loss function: ； X is the predicted key feature point probability map, Y is the true label map, and their sizes are (H / 8, W / 8, 65). Calculate the difference between X and Y, represents the number of all pixels in X, represents the sum of the probabilities of the true labels in the 65 channels of the i-th pixel in X, represents the probability of the jth channel of the i-th pixel in X, represents the sum of all probabilities of the 65 channels of the i-th pixel in X; ②Descriptor loss function: ； ； in, represents the perspective transformation matrix, It is the first The coordinates corresponding to the pixels, The first The coordinates corresponding to the pixels, the descriptor distance between the original image after perspective transformation and the matching image is expressed as , r represents the correspondence between two descriptors, 1 represents similarity, 0 represents dissimilarity, and are the descriptor sets on the original image and the matching image, is the set of all corresponding relations, and the descriptor similarity between matching pairs is calculated. express The number of all descriptors in , Used to measure the similarity between two descriptors. Indicates the similarity boundary threshold of the matching points, greater than Shows similarity, Indicates the similarity boundary threshold of non-matching points, which is less than Indicates dissimilarity; ③Semantic segmentation loss function: ； Among them, p represents the predicted value; q represents the true value, and They represent the predicted value and true value of the h-th pixel respectively.

9. The method for detecting key semantic feature points of an image according to claim 1, wherein: The accuracy in step 3 includes the accuracy of semantic segmentation and the accuracy of feature point detection; Semantic segmentation accuracy is evaluated by MloU and MPA. ； ； MloU represents the ratio of the intersection and union of the semantic label's true value and predicted value; MPA represents the proportion of correctly classified pixels in each class; Indicates the number of semantic categories; Representing semantic categories , Indicates other categories, Representation category The number of pixels predicted correctly, The actual category is But the number of pixels that are incorrectly predicted to be other classes, Indicates that it is actually another category but is incorrectly predicted as a category The number of pixels; The accuracy of feature point detection is evaluated by Repeatability, Matching Score, Mean Localization Error, and Nearest Neighbor Mean Average Precision: ； Repeatability measures the consistency of feature points under different perspectives, scales or image changes; Indicates the number of feature points detected at the same position in the two images, Represents the total number of feature points detected in the original image, represents the total number of feature points detected in the matching image, Indicates the minimum number of feature points detected in two images; ； Matching Score measures the accuracy of feature point detection and descriptor matching, where Indicates the number of correctly matched feature point pairs, Represents the total number of feature points detected in the matching image; ； MLE measures the average position error between the predicted key feature points and the true key feature points; S represents the number of all feature points in the original image, is the oth feature point position predicted by the model, is the oth true feature point position; Represents the Euclidean distance between the predicted key feature points and the true key feature points; ； ； ； ； NN mAP is an important indicator for evaluating the matching performance of feature descriptors and is used to measure the distinguishing ability of feature descriptors; TP is the number of samples correctly classified as positive examples, FP is the number of samples incorrectly classified as positive examples, and FN is the number of samples incorrectly classified as negative examples. AP calculates the average precision of each feature descriptor and determines whether two feature descriptors belong to the same category by setting different distance thresholds. TS is the number of different thresholds. is the current threshold index, is the number of descriptors in the prediction sample, is the current descriptor index.

Images