An efficient image feature point detection method and system based on deep learning

Abstract

The application discloses a kind of based on deep learning's image feature point efficient detection method and system, method includes the following steps: obtaining input image, pre-processing is carried out to input image, obtains single-channel gray scale chart;Encoding is carried out, and encoded feature map is obtained;Feature enhancement processing is carried out, and enhanced feature map is obtained;Feature is refined, and shared coding feature map is obtained;Detector decoding is carried out, and the feature map of first original resolution size, single-channel is obtained, and feature point is predicted;To shared coding feature map, descriptor decoding is carried out, and the feature map of second original resolution size, 256 channels is obtained, and descriptor is predicted;Multi-branch parameter fusion is carried out, and inference is carried out using fused parameter, to realize based on deep learning's image feature point efficient detection.The feature point detection method proposed in the application has higher precision and stronger stability in challenging scenarios compared to existing technologies.

Description

Technical Field

[0001] This application belongs to the field of computer image processing technology, specifically relating to an efficient method and system for detecting image feature points based on deep learning. Background Technology

[0002] Early computer vision tasks had simple requirements, and some classic feature point detection methods were designed to meet the needs of different application scenarios. However, as the application environment of current computer vision tasks becomes more and more complex, the adaptability of classic algorithms is getting lower and lower, especially in low-texture scenes with poor lighting conditions, where stability is insufficient and it is difficult to achieve a satisfactory level. To this end, deep learning-based methods are applied to the feature point extraction process to solve the problem of unstable feature point extraction when the lighting changes. The early exploration of deep learning-based feature point detection methods was not perfect, mainly due to the following problems: (1) The definition of feature points is unclear, and it is impossible to determine whether the labeled feature points are true feature points; (2) Manually labeling image data requires a lot of human resources, and it is difficult to obtain labeled image training data; (3) Image feature point labels automatically generated by unsupervised methods are always biased towards the edges, and the accuracy of the feature point definition needs to be verified. In addition to the careful design of the training dataset, the design of the network structure is equally important for improving the feature point detection performance. Currently, deep learning-based feature point detection methods mainly fall into the following categories: (1) combining handcrafted filters and learned filters for feature point detection, improving the repeatability of feature points by detecting key features at different scales. (2) upgrading the components of the network structure in existing methods, making the extracted local features geometrically invariant by associating information from low-level feature maps. (3) introducing an attention mechanism into the network structure of feature point detection to enhance feature representation capabilities. (4) redesigning the network structure of feature point detection using the multi-head self-attention mechanism Transformer to improve the accuracy of feature point detection. Although these methods improve the accuracy of feature point detection, they also increase the complexity of the network, resulting in unsatisfactory performance in terms of detection speed. It is not difficult to find that traditional feature point detection methods perform poorly in challenging environments such as poor lighting conditions, failing to meet the accuracy requirements. Deep learning-based feature point detection methods are stable and accurate, but there is a balance between efficiency and accuracy. Summary of the Invention

[0003] This application proposes an efficient image feature point detection method and system based on deep learning, which solves the problem of poor stability of feature point detection in challenging scenarios, while ensuring a balance between accuracy and efficiency of deep learning-based methods.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] A high-efficiency image feature point detection method based on deep learning includes the following steps:

[0006] S1: Obtain the input image, preprocess the input image to obtain a single-channel grayscale image;

[0007] S2: Based on a multi-branch large kernel convolutional structure, the single-channel grayscale image is encoded to obtain an encoded feature map;

[0008] S3: Based on a three-layer multi-branch stepwise convolutional structure, feature enhancement processing is performed on the encoded feature map to obtain an enhanced feature map;

[0009] S4: Based on a small multi-branch convolutional structure, refine the enhanced feature map to obtain a shared-encoded feature map;

[0010] S5: Decode the shared encoded feature map using a detector to obtain a single-channel feature map of the first original resolution size, and predict feature points;

[0011] S6: Decode the shared encoded feature map to obtain a feature map with a second original resolution size and 256 channels, and predict the descriptor;

[0012] S7: Perform multi-branch parameter fusion on the multi-branch large kernel convolutional structure of S2, the three-layer multi-branch stepwise convolutional structure of S3, and the small multi-branch convolutional structure of S4. Use the fused parameters to perform the prediction of S5 and S6 and the acquisition of feature maps, so as to achieve efficient detection of image feature points based on deep learning.

[0013] Preferably, the specific method for implementing S2 is as follows:

[0014] S2.1: Construct the multi-branch large-kernel convolutional structure at the head of the network backbone;

[0015] S2.2: Based on the asymmetric multi-branch large kernel convolutional structure, the single-channel grayscale image is encoded to obtain a channel feature map with significant discriminative power;

[0016] S2.3: Encode the single-channel grayscale image based on depthwise separable convolution to obtain a channel feature map with low discriminativeness, and perform channel-dimensional feature fusion with the channel feature map with high discriminativeness;

[0017] S2.4: Based on the symmetrical multi-branch large kernel convolutional structure, the first channel feature map obtained by feature fusion in S2.3 is processed to obtain an enhanced partial channel feature map;

[0018] S2.5: Based on the depthwise separable convolution, the first channel feature map obtained by feature fusion in S2.3 is encoded to obtain a first partial channel feature map, and then the channel-dimensional feature map is fused with the enhanced partial channel feature map to obtain the encoded feature map.

[0019] Preferably, the specific method for implementing S3 is as follows:

[0020] S3.1: Construct the three-layer multi-branch stepwise convolutional structure in the middle of the network backbone;

[0021] S3.2: Based on the asymmetric three-layer multi-branch stepwise convolutional structure, the encoded feature map is encoded again to obtain a feature map with refined features;

[0022] S3.3: Based on depthwise separable convolution, the encoded feature map is processed and fused with the refined feature map in the channel dimension.

[0023] S3.4: Based on the two-branch convolutional structure, the second channel feature map obtained by feature fusion in S3.3 is processed to obtain a partially enhanced feature map;

[0024] S3.5: Based on the depthwise separable convolution, process the second channel feature map obtained by feature fusion in S3.3 to obtain a second part of the channel feature map, and perform channel-dimensional feature fusion with the part of the enhanced feature map to obtain the enhanced feature map.

[0025] Preferably, the specific method for implementing S4 is as follows:

[0026] S4.1: Construct the small multi-branch convolutional structure at the tail of the network backbone;

[0027] S4.2: Based on the asymmetric small multi-branch convolutional structure, the enhanced feature map is processed to obtain a channel feature map containing rich features;

[0028] S4.3: Based on the depthwise separable convolution, the enhanced feature map is encoded to obtain a third part of the channel feature map, and the channel feature map containing rich features is fused with the channel dimension features to obtain the shared encoded feature map.

[0029] Preferably, the specific method for implementing S5 is as follows:

[0030] S5.1: Input the shared encoded feature map into the detector head, and perform channel mapping on the shared encoded feature map based on 1x1 convolution to obtain the first mapped feature map;

[0031] S5.2: Based on the first mapped feature map and the depth-to-space conversion, the detector is decoded to convert the 256-channel shared encoded feature map into a single-channel feature map of the first original resolution size.

[0032] S5.3: Based on the first original resolution size and single-channel feature map, predict the position of the feature points.

[0033] Preferably, the specific method for implementing S6 is as follows:

[0034] S6.1: Input the shared encoded feature map into the descriptor header, and perform channel mapping on the shared encoded feature map based on the 1x1 convolution to obtain the second mapped feature map;

[0035] S6.2: Perform bilinear interpolation on the second mapped feature map to obtain a feature map with the second original resolution size and 256 channels;

[0036] S6.3: Based on the feature map of the second original resolution size and 256 channels, predict the descriptor.

[0037] Preferably, the specific method for implementing S7 is as follows:

[0038] S7.1: After training, the multi-branch large kernel convolutional structure, the three-layer multi-branch step-by-step convolutional structure, and the small multi-branch convolutional structure are fused with their respective batch normalization layers to obtain three fused branches.

[0039] S7.2: The three fused branches are fused again to obtain a new convolutional structure with bias parameters, thereby realizing the multi-branch parameter fusion;

[0040] S7.3: Based on the new convolutional structure, a multi-branch network structure is obtained through the characteristics of multi-branch networks;

[0041] S7.4: Couple the multi-branch network structure to obtain a single-branch network structure to realize the inference.

[0042] Preferably, the multi-branch large-kernel convolutional structure includes a first part and a second part;

[0043] The first part includes: 5*5, 1*5, 5*1 multi-branch asymmetric convolutional structures and depthwise separable convolutional structures with fewer computational parameters;

[0044] The second part includes: 5*5, 3*3, and 1*1 multi-branch symmetric convolutional structures.

[0045] A high-efficiency image feature point detection system based on deep learning includes: a preprocessing module, an encoding module, an enhancement module, a thinning module, a detector decoding module, a descriptor decoding module, and a fusion module;

[0046] The preprocessing module is used to obtain the input image, preprocess the input image, and obtain a single-channel grayscale image.

[0047] The encoding module is used to encode the single-channel grayscale image to obtain an encoded feature map;

[0048] The enhancement module is used to perform feature enhancement processing on the encoded feature map to obtain an enhanced feature map;

[0049] The refinement module is used to refine the enhanced feature map to obtain a shared-encoded feature map.

[0050] The detector decoding module is used to perform detector decoding on the shared encoded feature map to obtain a single-channel feature map of the first original resolution size, and to predict feature points;

[0051] The descriptor decoding module is used to perform descriptor decoding on the shared encoded feature map to obtain a feature map with a second original resolution size and 256 channels, and to predict the descriptor.

[0052] The fusion module is used to perform multi-branch parameter fusion on the multi-branch large kernel convolutional structure, the three-layer multi-branch stepwise convolutional structure, and the small multi-branch convolutional structure. The fused parameters are then used to perform prediction and feature map acquisition by the detector decoding module and the descriptor decoding module, thereby achieving efficient detection of image feature points based on deep learning.

[0053] The beneficial effects of this application are as follows: This application designs different types of multi-branch stepwise convolutional modules, significantly improving the feature extraction capability of the network by utilizing asymmetric convolutional structures and ordinary multi-branch convolutional structures. Compared with existing technologies, the feature point detection method proposed in this invention has higher accuracy and stronger stability in challenging scenarios. This invention designs an efficient stepwise convolutional strategy, which effectively reduces the computational parameters and computational complexity of the network and improves the network's operating efficiency compared with existing deep learning-based feature point detection methods. A large-kernel convolutional structure is used in the network backbone head, increasing the receptive field of the initial encoding, allowing the network to retain more original feature information and reducing the impact of deep feature information loss. This application exhibits more satisfactory performance in other computer vision tasks based on point features. This application has broad application potential and practical value. Attached Figure Description

[0054] To more clearly illustrate the technical solutions of this application, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of the efficient image feature point detection method based on deep learning, as described in Embodiment 1 of this application.

[0056] Figure 2 This is a network structure diagram of Embodiment 1 of this application;

[0057] Figure 3 Figure A shows a multi-branch step-by-step convolutional structure diagram of an efficient distributed convolutional strategy according to Embodiment 2 of this application. Figure B shows a large kernel convolutional structure; Figure C shows a multi-branch step-by-step convolutional structure; and Figure C shows a small multi-branch convolutional structure at the tail of the network backbone.

[0058] Figure 4 This is a structural diagram of the reparameterized large kernel convolution module in Embodiment 2 of this application;

[0059] Figure 5 This is a flowchart illustrating the overall implementation of Embodiment 3 of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Example 1:

[0063] like Figure 1-2 As shown, the network structure of this application is a distributed multi-task framework with a shared coding structure. Setting up a shared coding structure enables parameter sharing between the detector and descriptors, effectively reducing the overall computational parameters of the network. Furthermore, the distributed structure allows for simultaneous training of both the detector and descriptors.

[0064] A high-efficiency image feature point detection method based on deep learning includes the following steps:

[0065] S1: Obtain the input image, preprocess the input image, and obtain a single-channel grayscale image;

[0066] S2: Based on a multi-branch large kernel convolutional structure, a single-channel grayscale image is encoded to obtain an encoded feature map;

[0067] S3: Based on a three-layer multi-branch stepwise convolutional structure, feature enhancement processing is performed on the encoded feature map to obtain an enhanced feature map;

[0068] S4: Based on a small multi-branch convolutional structure, the enhanced feature map is refined to obtain a shared encoded feature map;

[0069] S5: Decode the shared encoded feature map using a detector to obtain a single-channel feature map at the first original resolution size, and predict feature points;

[0070] S6: Decode the shared encoded feature map to obtain a second original resolution feature map with 256 channels, and predict the descriptor;

[0071] S7: Multi-branch parameter fusion is performed on the multi-branch large kernel convolutional structure of S2, the three-layer multi-branch stepwise convolutional structure of S3, and the small multi-branch convolutional structure of S4. The fused parameters are used to predict S5 and S6 and obtain feature maps, so as to achieve efficient detection of image feature points based on deep learning.

[0072] Specifically, the method to implement S2 is as follows: input the single-channel grayscale image into the large-kernel convolutional structure of the network backbone head for encoding.

[0073] S2.1: Construct a multi-branch, large-kernel convolutional structure at the head of the network backbone;

[0074] S2.2: Based on an asymmetric multi-branch large kernel convolutional structure, a single-channel grayscale image is encoded to obtain a channel feature map with significant discriminative power.

[0075] S2.3: Encoding a single-channel grayscale image based on depthwise separable convolution to obtain a channel feature map with low discriminativeness, and then performing channel-dimensional feature fusion with the channel feature map with high discriminativeness;

[0076] S2.4: Based on a symmetrical multi-branch large kernel convolutional structure, the first channel feature map obtained by feature fusion in S2.3 is processed to obtain an enhanced partial channel feature map;

[0077] S2.5: Based on depthwise separable convolution, the first channel feature map obtained by feature fusion in S2.3 is encoded to obtain the first part of the channel feature map. This part of the channel feature map is then fused with the enhanced partial channel feature map to obtain the encoded feature map. S2 uses a large kernel convolution structure in the network backbone head, which increases the receptive field of the initial encoding, allowing the network to retain more original feature information and reducing the impact of loss of deep feature information.

[0078] Specifically, the method to implement S3 is as follows: input the feature map encoded in the head of the network backbone (i.e., the encoded feature map) into the three-layer multi-branch stepwise convolutional structure in the middle of the network backbone to achieve feature enhancement processing of the feature map.

[0079] S3.1: Construct a three-layer multi-branch stepwise convolutional structure in the middle of the network backbone;

[0080] S3.2: Based on an asymmetric three-layer multi-branch stepwise convolutional structure, the encoded feature map is encoded again to obtain a feature map with refined features;

[0081] S3.3: Based on depthwise separable convolution, the encoded feature map is processed and fused with the feature map of feature refinement in the channel dimension;

[0082] S3.4: Based on the two-branch convolutional structure, the second channel feature map obtained by feature fusion in S3.3 is processed to obtain a partially enhanced feature map;

[0083] S3.5: Based on depthwise separable convolution, the second channel feature map obtained by feature fusion in S3.3 is encoded to obtain the second part of the channel feature map, and then the channel dimension feature is fused with the partial enhanced feature map to obtain the enhanced feature map.

[0084] Specifically, the method to implement S4 is as follows: input the feature map (i.e. the enhanced feature map) processed in the middle of the network backbone into a small multi-branch convolutional structure at the tail of the network backbone to further refine the features and complete the shared encoding of the feature map.

[0085] S4.1: Construct a small multi-branch convolutional structure at the tail of the network backbone;

[0086] S4.2: Based on an asymmetric small multi-branch convolutional structure, the enhanced feature map is processed to obtain a channel feature map containing rich features;

[0087] S4.3: Based on depthwise separable convolution, the enhanced feature map is encoded to obtain the third part of the channel feature map, and then the channel feature map containing rich features is fused with the channel dimension features to obtain a shared encoded feature map.

[0088] Specifically, the method for implementing S5 is as follows: input the shared encoded feature map into the head of the detector module (i.e., the detector head), and output the first original resolution size, single-channel feature map after the detector decodes it, and predict feature points in the feature map.

[0089] S5.1: Input the shared encoded feature map into the detector head, and perform channel mapping on the shared encoded feature map based on 1x1 convolution to obtain the first mapped feature map;

[0090] S5.2: Based on the feature map after the first mapping and the transformation from depth to space (i.e., depth-space module), perform detector decoding to convert the 256-channel shared encoded feature map into a single-channel feature map of the first original resolution size;

[0091] S5.3: Based on the feature map of the first original resolution size and single channel, predict the location of feature points.

[0092] Specifically, the method for implementing S6 is as follows: input the shared encoded feature map into the header of the descriptor module (i.e., the descriptor header), output the second original resolution size and 256 channels feature map after the descriptor is decoded, and predict the descriptor in the feature map.

[0093] S6.1: Input the shared encoded feature map into the descriptor header, perform channel mapping on the shared encoded feature map based on 1x1 convolution, and obtain the second mapped feature map;

[0094] S6.2: Perform bilinear interpolation (i.e., linear interpolation module) on the second mapped feature map to obtain a feature map with the second original resolution size and 256 channels;

[0095] S6.3: Predict descriptors based on the feature map with a second original resolution and 256 channels.

[0096] Specifically, the method to implement S7 is as follows: after training, all multi-branch structures in the above steps are fused using the reparameter method, and the fused parameters are used for inference to reduce the running time during inference.

[0097] S7.1: After training, the multi-branch large kernel convolutional structure, the three-layer multi-branch step-by-step convolutional structure, and the small multi-branch convolutional structure are fused with their respective batch normalization layers to obtain three fused branches.

[0098] S7.2: The three fused branches are fused again to obtain a new convolutional structure with bias parameters, thus achieving multi-branch parameter fusion;

[0099] S7.3: Based on the new convolutional structure, a multi-branch network structure is obtained through the characteristics of multi-branch networks;

[0100] S7.4: Couple the multi-branch network structure to obtain a single-branch network structure to realize the inference.

[0101] The network structure designed in this application utilizes a more flexible asymmetric convolutional structure to extract features, giving the network a stronger feature point detection capability. The designed network structure is a multi-branch network structure, fully utilizing the feature information encoded by different receptive fields, thus enabling more reliable feature point detection. The designed network structure employs a step-by-step convolution strategy, using depthwise separable convolutions with fewer computational parameters to generate similar channel feature maps, improving the network's operational efficiency. After training, S7 fuses the multi-branch parameters through structural reparameterization, coupling the multi-branch network structure into a common single-branch structure during inference, achieving even higher inference efficiency.

[0102] Example 2:

[0103] like Figure 3-4 As shown, this application also provides an efficient distributed convolution strategy:

[0104] The three different types of multi-branch convolutional structures designed in this application constitute the network backbone of the shared coding, specifically including the multi-branch large kernel convolutional structure at the head of the network backbone, the three-layer multi-branch step-by-step convolutional structure in the middle of the network backbone, and the small multi-branch convolutional structure at the tail of the network backbone.

[0105] The multi-branch large-kernel convolutional structure consists of a first part and a second part;

[0106] The first part includes: 5*5, 1*5, and 5*1 multi-branch asymmetric convolutional structures and depthwise separable convolutional structures with fewer computational parameters;

[0107] The second part includes: 5*5, 3*3, and 1*1 multi-branch symmetric convolutional structures.

[0108] Specifically, Figure 3A is a large-kernel convolutional structure, consisting of two parts, each composed of a multi-branch convolutional structure and a depthwise separable convolutional structure. The first layer of the first part is a multi-branch asymmetric convolutional structure composed of 5x5, 1x5, and 5x1 branches. Asymmetric convolutional structures offer high flexibility and stronger feature extraction capabilities. The second layer of the first part is a depthwise separable convolution with fewer computational parameters, primarily used to reduce encoding time and improve encoding efficiency. The first layer of the second part consists of a multi-branch convolutional structure composed of 5x5, 3x3, and 1x1 branches. Convolutional structures with different receptive fields extract different feature information; the fusion of multiple branches enriches the feature information. In particular, the large-kernel convolution setting reduces the impact of feature information loss in deep networks.

[0109] Figure 3 B is a multi-branch stepwise convolutional structure (i.e., a three-layer multi-branch stepwise convolutional structure) in the middle of the network backbone. Its overall design is similar to the large-kernel convolutional structure, but the difference lies in that the first layer of the first part uses a 3*3, 1*3, and 3*1 multi-branch asymmetric convolutional structure, while the first layer of the second part only has two convolutional branches: 3*3 and 1*1. The middle part of the network backbone is a crucial bridge for feature information transmission, requiring both information integrity and consideration of the overall computational complexity of the network. Therefore, optimizations and adjustments were made based on the large-kernel convolutional structure.

[0110] Figure 3 C is a small, multi-branch convolutional structure at the tail of the network backbone, identical to the first part of the network backbone. Located at the end of the shared coding structure, it is primarily responsible for further integrating and refining the features output from the middle of the network backbone, facilitating decoding by the detector and descriptors, and improving the network's prediction accuracy.

[0111] It is important to note that all the multi-branch convolutional structures mentioned above will be transformed into a new set of parameters for the convolutional structure after training through a structure reparameterization method. The multi-branch structure is then coupled into a regular single-branch forward propagation structure. The specific transformation process is as follows: Figure 4 As shown, the three convolutional structures with different branches are first fused with their respective BN (batch normalization) layers. Then, the parameters of the three fused branches are fused again, transforming them into a new convolutional structure with bias parameters. During training, the characteristics of multi-branch networks are utilized to obtain a set of multi-branch network parameters. During inference, the multi-branch network structure is coupled into a more efficient single-branch network structure. This approach not only utilizes the stronger feature extraction capabilities of multi-branch network structures but also achieves the inference efficiency of single-branch structures.

[0112] Example 3:

[0113] To achieve the aforementioned efficient image feature point detection method based on deep learning, this invention trains a feature point detection network in a self-supervised manner. The specific implementation process of this invention is as follows: Figure 5 As shown, this avoids the significant manpower required for manually labeling datasets. First, pre-training: A virtual dataset with a 3D geometric model containing defined corner positions is generated. The feature point detection network is trained using this virtual dataset to predict feature points. When the number of iterations is greater than N, the network weights with the best feature point prediction results are output and used to label the real image dataset. When the number of iterations is less than or equal to N, the process returns to generating the virtual dataset with defined corner positions. Next, self-supervised training: The feature point detection network is retrained using the labeled real image dataset to predict feature points. When the number of iterations is greater than N, the network weights with the best feature point prediction results are output and used to load the reparameterized feature point detection network. When the number of iterations is less than or equal to N, the process returns to labeling the real image dataset. Once the optimal network model is obtained after training, the network weights with the best feature point prediction results are output. Finally, inference testing: The network model with the optimal weights is reparameterized to obtain the reparameterized feature point detection network. Feature point prediction is then performed to obtain the final inference model, and test images are loaded for inference testing.

[0114] Example 4:

[0115] To realize the efficient image feature point detection method based on deep learning proposed in this application, an efficient image feature point detection system based on deep learning is proposed, including: a preprocessing module, an encoding module, an enhancement module, a thinning module, a detector decoding module, a descriptor decoding module, and a fusion module;

[0116] The preprocessing module is used to obtain the input image, preprocess the input image, and obtain a single-channel grayscale image.

[0117] The encoding module is used to encode a single-channel grayscale image to obtain an encoded feature map.

[0118] The enhancement module is used to perform feature enhancement processing on the encoded feature map to obtain an enhanced feature map;

[0119] The refinement module is used to refine the enhanced feature map to obtain a shared-encoded feature map;

[0120] The detector decoding module is used to decode the shared encoded feature map to obtain a single-channel feature map of the first original resolution size and to predict feature points.

[0121] The descriptor decoding module is used to decode the shared encoded feature map to obtain a second original resolution size, 256 channels of feature map, and predict the descriptor.

[0122] The fusion module is used to fuse multi-branch parameters of multi-branch large kernel convolutional structures, three-layer multi-branch stepwise convolutional structures, and small multi-branch convolutional structures. The fused parameters are then used for prediction and feature map acquisition by the detector decoding module and descriptor decoding module, achieving efficient detection of image feature points based on deep learning.

[0123] Specifically, the encoding module: inputs the single-channel grayscale image into the large-kernel convolutional structure of the network backbone head for encoding.

[0124] At the head of the network backbone, construct a multi-branch large-kernel convolutional structure;

[0125] Based on an asymmetric multi-branch large kernel convolutional structure, a single-channel grayscale image is encoded to obtain a channel feature map with significant discriminative power.

[0126] Encoding a single-channel grayscale image using depthwise separable convolution yields a channel feature map with low discriminativeness, which is then fused with a channel feature map that has high discriminativeness in the channel dimension.

[0127] Based on a symmetrical multi-branch large kernel convolutional structure, the first channel feature map after feature fusion is processed to obtain an enhanced partial channel feature map;

[0128] Based on depthwise separable convolution, the first channel feature map obtained after feature fusion is encoded to obtain a first partial channel feature map. This first partial channel feature map is then fused with the enhanced partial channel feature map to obtain an encoded feature map. The network backbone head uses a large kernel convolution structure, which increases the receptive field of the initial encoding, allowing the network to retain more original feature information and reducing the impact of loss of deep feature information.

[0129] Specifically, the enhancement module inputs the feature map encoded in the head of the network backbone (i.e., the encoded feature map) into the three-layer multi-branch stepwise convolutional structure in the middle of the network backbone to achieve feature enhancement processing of the feature map.

[0130] In the middle of the network backbone, a three-layer multi-branch stepwise convolutional structure is constructed;

[0131] Based on an asymmetric three-layer multi-branch stepwise convolutional structure, the encoded feature map is encoded again to obtain a feature map with refined features.

[0132] Based on depthwise separable convolution, the encoded feature map is processed and fused with the feature map of feature refinement in the channel dimension.

[0133] Based on a two-branch convolutional structure, the second channel feature map after feature fusion is processed to obtain a partially enhanced feature map.

[0134] Based on depthwise separable convolution, the second channel feature map after feature fusion is processed to obtain the second part of the channel feature map, and then fused with the partial enhanced feature map in the channel dimension to obtain the enhanced feature map.

[0135] Specifically, the refinement module: inputs the feature maps processed in the middle of the network backbone (i.e., the enhanced feature maps) into a small multi-branch convolutional structure at the tail of the network backbone to further refine the features and complete the shared encoding of the feature maps.

[0136] At the tail of the network backbone, a small multi-branch convolutional structure is constructed;

[0137] Based on an asymmetric small multi-branch convolutional structure, the enhanced feature map is processed to obtain a channel feature map containing rich features.

[0138] Based on depthwise separable convolution, the enhanced feature map is encoded to obtain a third part of the channel feature map, and then the channel feature map containing rich features is fused with the channel dimension features to obtain a shared encoded feature map.

[0139] Specifically, the detector decoding module: inputs the shared encoded feature map into the head of the detector module (i.e., the detector head), and outputs a single-channel feature map of the first original resolution size after the detector decodes it, and predicts feature points in this feature map.

[0140] The shared encoded feature map is input into the detector head, and channel mapping is performed on the shared encoded feature map based on 1x1 convolution to obtain the first mapped feature map.

[0141] Based on the feature map after the first mapping and the transformation from depth to space (i.e., depth-space module), the detector decodes and converts the 256-channel shared encoded feature map into a single-channel feature map of the first original resolution size.

[0142] Based on the feature map of the first original resolution size and single channel, the location of feature points is predicted.

[0143] Specifically, the descriptor decoding module: inputs the shared encoded feature map into the header of the descriptor module (i.e., the descriptor header), outputs a feature map of the second original resolution size and 256 channels after the descriptor is decoded, and predicts the descriptor in this feature map;

[0144] The shared encoded feature map is input into the descriptor header. Based on a 1x1 convolution, channel mapping is performed on the shared encoded feature map to obtain the second mapped feature map.

[0145] The second mapped feature map is subjected to bilinear interpolation (i.e., linear interpolation module) to obtain a second original resolution feature map with 256 channels.

[0146] Descriptor prediction is performed based on a feature map with a second original resolution and 256 channels.

[0147] Specifically, the fusion module: After training, the parameters of all multi-branch structures in the above steps are fused using the reparameter method, and the fused parameters are used for inference to reduce the running time during inference.

[0148] After training, the multi-branch large kernel convolutional structure, the three-layer multi-branch step-by-step convolutional structure, and the small multi-branch convolutional structure are fused with their respective batch normalization layers to obtain three fused branches.

[0149] The three fused branches are then fused again to obtain a new convolutional structure with bias parameters, thus achieving multi-branch parameter fusion.

[0150] Based on the new convolutional structure, a multi-branch network structure is obtained by leveraging the characteristics of multi-branch networks;

[0151] By coupling multi-branch network structures, a single-branch network structure can be obtained to achieve inference.

[0152] The embodiments described above are merely preferred embodiments of this application and are not intended to limit the scope of this application. Any modifications and improvements made to the technical solutions of this application by those skilled in the art without departing from the spirit of this application shall fall within the protection scope defined by the claims of this application.

Claims

1. A method for efficient image feature point detection based on deep learning, characterized in that, Includes the following steps: S1: Obtain the input image, preprocess the input image to obtain a single-channel grayscale image; S2: Based on a multi-branch large kernel convolutional structure, the single-channel grayscale image is encoded to obtain an encoded feature map; S3: Based on a three-layer multi-branch stepwise convolutional structure, feature enhancement processing is performed on the encoded feature map to obtain an enhanced feature map; S4: Based on a small multi-branch convolutional structure, refine the enhanced feature map to obtain a shared-encoded feature map; S5: Decode the shared encoded feature map using a detector to obtain a single-channel feature map of the first original resolution size, and predict feature points; S6: Decode the shared encoded feature map to obtain a feature map with a second original resolution size and 256 channels, and predict the descriptor; S7: Perform multi-branch parameter fusion on the multi-branch large kernel convolutional structure of S2, the three-layer multi-branch stepwise convolutional structure of S3, and the small multi-branch convolutional structure of S4. Use the fused parameters to perform prediction of S5 and S6 and feature map acquisition, so as to achieve efficient detection of image feature points based on deep learning. The specific method for implementing S2 is as follows: S2.1: Construct the multi-branch large-kernel convolutional structure at the head of the network backbone; S2.2: Based on the asymmetric multi-branch large kernel convolutional structure, the single-channel grayscale image is encoded to obtain a channel feature map with significant discriminative power; S2.3: Encode the single-channel grayscale image based on depthwise separable convolution to obtain a channel feature map with low discriminativeness, and perform channel-dimensional feature fusion with the channel feature map with high discriminativeness; S2.4: Based on the symmetrical multi-branch large kernel convolutional structure, the first channel feature map obtained by feature fusion in S2.3 is processed to obtain an enhanced partial channel feature map; S2.5: Based on the depthwise separable convolution, the first channel feature map obtained by feature fusion in S2.3 is encoded to obtain a first partial channel feature map, and channel-dimensional feature fusion is performed with the enhanced partial channel feature map to obtain the encoded feature map; The specific method for implementing S3 is as follows: S3.1: Construct the three-layer multi-branch stepwise convolutional structure in the middle of the network backbone; S3.2: Based on the asymmetric three-layer multi-branch stepwise convolutional structure, the encoded feature map is encoded again to obtain a feature map with refined features; S3.3: Based on depthwise separable convolution, the encoded feature map is processed and fused with the refined feature map in the channel dimension. S3.4: Based on the two-branch convolutional structure, the second channel feature map obtained by feature fusion in S3.3 is processed to obtain a partially enhanced feature map; S3.5: Based on the depthwise separable convolution, process the second channel feature map obtained by feature fusion in S3.3 to obtain a second part channel feature map, and perform channel-dimensional feature fusion with the part enhanced feature map to obtain the enhanced feature map; The specific method for implementing S4 is as follows: S4.1: Construct the small multi-branch convolutional structure at the tail of the network backbone; S4.2: Based on the asymmetric small multi-branch convolutional structure, the enhanced feature map is processed to obtain a channel feature map containing rich features; S4.3: Based on the depthwise separable convolution, the enhanced feature map is encoded to obtain a third part of the channel feature map, and the channel feature map containing rich features is fused with the channel dimension feature map to obtain the shared encoded feature map.

2. The efficient image feature point detection method based on deep learning according to claim 1, characterized in that, The specific method for implementing S5 is as follows: S5.1: Input the shared encoded feature map into the detector head, and perform channel mapping on the shared encoded feature map based on 1x1 convolution to obtain the first mapped feature map; S5.2: Based on the first mapped feature map and the depth-to-space conversion, the detector is decoded to convert the 256-channel shared encoded feature map into a single-channel feature map of the first original resolution size. S5.3: Based on the first original resolution size and single-channel feature map, predict the position of the feature points.

3. The efficient image feature point detection method based on deep learning according to claim 2, characterized in that, The specific method for implementing S6 is as follows: S6.1: Input the shared encoded feature map into the descriptor header, and perform channel mapping on the shared encoded feature map based on the 1x1 convolution to obtain the second mapped feature map; S6.2: Perform bilinear interpolation on the second mapped feature map to obtain a feature map with the second original resolution size and 256 channels; S6.3: Based on the feature map of the second original resolution size and 256 channels, predict the descriptor.

4. The efficient image feature point detection method based on deep learning according to claim 3, characterized in that, The specific method for implementing S7 is as follows: S7.1: After training, the multi-branch large kernel convolutional structure, the three-layer multi-branch step-by-step convolutional structure, and the small multi-branch convolutional structure are fused with their respective batch normalization layers to obtain three fused branches. S7.2: The three fused branches are fused again to obtain a new convolutional structure with bias parameters, thereby realizing the multi-branch parameter fusion; S7.3: Based on the new convolutional structure, a multi-branch network structure is obtained through the characteristics of multi-branch networks; S7.4: Couple the multi-branch network structure to obtain a single-branch network structure for inference.

5. The efficient image feature point detection method based on deep learning according to claim 1, characterized in that, The multi-branch large-kernel convolutional structure includes a first part and a second part; The first part includes: 5*5, 1*5, 5*1 multi-branch asymmetric convolutional structures and depthwise separable convolutional structures with fewer computational parameters; The second part includes: 5*5, 3*3, and 1*1 multi-branch symmetric convolutional structures.

6. A high-efficiency image feature point detection system based on deep learning, used to implement the method according to any one of claims 1-5, characterized in that, include: The module includes a preprocessing module, an encoding module, an enhancement module, a refinement module, a detector decoding module, a descriptor decoding module, and a fusion module. The preprocessing module is used to obtain the input image, preprocess the input image, and obtain a single-channel grayscale image. The encoding module is used to encode the single-channel grayscale image to obtain an encoded feature map; The enhancement module is used to perform feature enhancement processing on the encoded feature map to obtain an enhanced feature map; The refinement module is used to refine the enhanced feature map to obtain a shared-encoded feature map. The detector decoding module is used to perform detector decoding on the shared encoded feature map to obtain a single-channel feature map of the first original resolution size, and to predict feature points; The descriptor decoding module is used to perform descriptor decoding on the shared encoded feature map to obtain a feature map with a second original resolution size and 256 channels, and to predict the descriptor. The fusion module is used to perform multi-branch parameter fusion on the multi-branch large kernel convolutional structure, the three-layer multi-branch stepwise convolutional structure, and the small multi-branch convolutional structure. The fused parameters are then used to perform prediction and feature map acquisition by the detector decoding module and the descriptor decoding module, thereby achieving efficient detection of image feature points based on deep learning.

Images