Image matching method based on RepVGG lightweight network

Abstract

An image matching method based on a lightweight RepVGG network comprises the following steps: Step S1, constructing a lightweight feature extraction network to extract multi-channel feature maps. Step S2, extracting feature points and calculating descriptors from the multi-channel feature maps. Step S3, performing feature matching using the FLANN algorithm. Step S4, using an improved RANSAC algorithm to eliminate mismatches and obtain the final matching pairs. This invention is a feature matching method based on deep learning, proposing an image matching method based on a lightweight RepVGG network. Based on the RepVGG network and combined with the characteristics of the GCNv2 network, a lightweight feature matching network with a simple structure and high readability is designed for the field of image matching, saving computational power while reducing the network training burden and making it easier to train and deploy.

Description

Technical Field

[0001] This invention belongs to the technical field of image matching methods, and specifically relates to an image matching method based on the RepVGG lightweight network. Background Art

[0002] Image matching is an important research area in the field of image vision. Essentially, it's a technique for finding corresponding positional relationships among multiple images. Based on this, the relative motion states between adjacent frames can be calculated, or point clouds can be computed. It is a fundamental research area in visual navigation, 3D reconstruction, and visual SLAM. Furthermore, because visual sensors can acquire rich texture information and possess powerful scene recognition capabilities, the approach of solving related problems from a vision-based perspective is favored by researchers. Therefore, researching a real-time image matching method has significant application value in the field of vision.

[0003] Recent literature and experimental results indicate that deep learning-based feature matching methods are quite popular. For example, the LoFTR algorithm (Detector-Free Local Feature Matching with Transformers) adopts an end-to-end matching approach, taking an image as input and directly outputting matching pairs; feature map extraction and subsequent matching are completed through optimizing the transmission problem, all within the network. However, this method is time-consuming in the feature point extraction and matching stages, and still lacks real-time performance; furthermore, it is prone to information loss when encountering occlusion, which can be fatal in some application scenarios.

[0004] Furthermore, the GCNv2 algorithm (GCNv2: Efficient Correspondence Prediction for Real-Time SLAM) has been proven to run in real-time on a GPU. This method extracts feature points from an image, describes them, and calculates descriptors by binarizing them or calculating the surrounding vectors of the features, then matches them. Although this method can run in real-time on a GPU, its relatively simple network structure and insufficient feature extraction depth lead to low accuracy and insufficient number of extracted features under conditions such as low texture, thus affecting the matching results. Therefore, this algorithm still has room for improvement and optimization in terms of accuracy.

[0005] In summary, a lightweight network needs to be designed to address the problems in existing technologies. This network needs to distinguish most features in an image and complete the matching. At the same time, the network structure must meet the requirements of reducing computational power and time consumption without excessively sacrificing algorithm accuracy, which is a challenging task. Summary of the Invention

[0006] In view of the technical problems existing in the background technology, the present invention provides an image matching method based on the RepVGG lightweight network. Based on the RepVGG network and combined with the characteristics of the GCNv2 network, a lightweight feature matching network with simple structure and high readability is designed for the field of image matching, which saves computing power and reduces the training burden of the network.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] An image matching method based on the RepVGG lightweight network, comprising the following steps:

[0009] Step S1: Construct a lightweight feature extraction network to extract multi-channel feature maps.

[0010] Step S2: Extract feature points and calculate descriptors from the multi-channel feature map.

[0011] Step S3: Use the FLANN algorithm for feature matching.

[0012] Step S4: Use the improved RANSAC algorithm to remove mismatches and obtain the final matching pairs.

[0013] Preferably, in step S1, a downsampling network with RepVGG as the backbone is constructed, and the feature maps are sampled using the following method:

[0014] S101. Construct a RepVGG-block: Construct a multi-branch structure. Input P undergoes a 3x3 convolution and a BN normalization to obtain P1, with network parameters N1; simultaneously, image P undergoes a 1x1 convolution and a BN normalization to obtain P2, with network parameters N2; furthermore, P undergoes a BN normalization to obtain the identity mapping P3, with network parameters N3, denoted as the Identity layer; in the Identity layer branch, set the judgment A.

[0015] Judgment A: Only when the RepVGG-block module does not change the number of feature channels and the convolution stride is set to 1, is the RepVGG-block a normal structure and remains unchanged; otherwise, delete the Identity layer branch here, and then perform downsampling; add the results of P1, P2, and P3; then activate using the GELU function.

[0016] When the network is in the training phase, the above RepVGG-block structure remains unchanged; when the network is in the deployment model inference phase, the network parameters of the two RepVGG-blocks in judgment A are reparameterized, and the multi-branch network structure is merged into a single-path network structure.

[0017] S102. Construct a lightweight downsampling network structure using RepVGG-block. Stack the RepVGG-block module into 8 layers. The number of input channels of the RepVGG-block is defined as the number of input feature channels, and the number of output channels is defined as [32, 32, 64, 64, 128, 128, 256, 256]. Among them, layers 1, 3, 5, and 7 are downsampling layers; layers 2, 4, 6, and 8 are normal RepVGG-block structures. After activation by the GELU function in layer 1, a CBAM module is added; after activation by the GELU function in layer 8, a CBAM module is added, and the network construction is completed. Input the image into the network to obtain a feature map with 256 channels, defined as... .

[0018] Preferably, the decomposition steps of step S2 are as follows:

[0019] S201. Divide the feature F into two parts, F1 and F2. Activate F1 using the sigmoid function to obtain the predicted probability map, denoted as convf. Upsample convf using PixelShuffle to change the pixel size of the probability map, making convf a single-channel image and restoring it to the original input image pixel size, denoted as ConvF. Perform non-maximum suppression on ConvF, denoted as NMS-F. Determine the feature point coordinates of the original input image based on NMS-F, denoted as POS-F.

[0020] S202. Calculate the norm of the first channel of F2, divide F2 by this norm to obtain a low-resolution dense feature map, denoted as convd; based on POS-F, perform bilinear sampling on the corresponding feature vectors in convd, followed by binarization. Extract the binary descriptor corresponding to each feature point, as shown in the following formula:

[0021]

[0022] Where z is the coordinate of the feature map, and f is the compressed-scale feature map extracted from convd;

[0023] Combined with binarized descriptors Using the feature point coordinates POS-F, extract the final feature points;

[0024] S203: Configure the AdamW optimizer to optimize the binary descriptor, which is dominated by 0s and 1s. The cross-loss entropy function is calculated with an initial learning rate of 0.0001 and a maximum number of iterations of 100 epochs. The learning rate is multiplied by a coefficient of 0.55 every 25 epochs.

[0025] Preferably, in step S3, when performing feature matching using the FLANN algorithm, the POS-F obtained in step S201 and the corresponding features obtained in S202 are... Save the data as a dictionary and use the FLANN algorithm to perform feature matching.

[0026] Preferably, step S4 is performed as follows:

[0027] Let the set of matching point pairs obtained in step S3 be denoted as The matched pairs are arranged in descending order of quality. The first 60% of the pairs are selected and denoted as set M. When set M contains fewer than 100 elements, the set is... Let M be the set of points. Then, the RANSAC algorithm is used to randomly select 4 point pairs from set M to calculate the homography transformation matrix H, and the inliers are filtered to remove mismatches. The feature point quality evaluation index is the NMS-F value of the predicted probability map in S2, and the matching point pair quality is the sum of the NMS-F values ​​of the feature point pairs in the matching image.

[0028] Preferably, the structural reparameterization operation is as follows:

[0029] The network parameters N2 of the 1x1 convolutional layer and BN layer, and the network parameters N3 of the Identity layer are used for parameter inference and fused with the network parameters N1 of the 3x3 convolutional layer and BN layer. At the same time, the fused network structure is replaced with the network structure of the 3x3 convolutional layer, and then the convolutional layers are stacked to form a single-path structure for inference and prediction.

[0030] For N2, the 1x1 convolutional kernel is padded to a 3x3 form through the Conv2D convolutional layer; the result is added to N1, and the result after addition is convolved with 1x1, that is, the parameters of the 1x1 convolutional layer are transformed into the parameters of the 3x3 convolutional layer network.

[0031] For N3, the Conv2D convolutional layer sets the kernel parameter of the current channel to 1 and the kernel parameters of the rest to 0, padding it to a 3x3 convolutional form. The result is added to N1, and a 1x1 convolution is performed on the result to transform it into the network layer parameters of a 3x3 convolutional layer.

[0032] Preferably, in step S1, the CBAM module is connected by spatial attention and channel attention. During the convolution feature extraction process, the original image is transformed into a multi-channel feature map, and weights are assigned to the features, making the features easier for the computer to recognize.

[0033] This patent can achieve the following beneficial effects:

[0034] 1. This invention proposes a feature matching method based on deep learning, specifically an image matching method using a lightweight RepVGG network. Based on the RepVGG network and incorporating the characteristics of the GCNv2 network, a simple, highly readable, and lightweight feature matching network is designed for the image matching field. This saves computational resources, reduces the network training burden, and makes it easier to train and deploy.

[0035] 2. Based on the quality of the matched point pairs, this invention optimizes the point set in the RANSAC algorithm, shortening the original computation time, which is important for spatial vision tasks such as 3D reconstruction and navigation. Attached Figure Description

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0037] Figure 1 This is a flowchart of the present invention;

[0038] Figure 2 This is a network structure diagram of the present invention;

[0039] Figure 3 Supplementary diagrams for the network modules of this invention;

[0040] Figure 4 This is a diagram of the CBAM structure. Detailed Implementation

[0041] Example 1:

[0042] Preferred solutions include Figures 1 to 4 As shown, an image matching method based on the RepVGG lightweight network comprises the following steps:

[0043] Step S1: Construct a lightweight feature extraction network to extract multi-channel feature maps. The steps are broken down as follows:

[0044] S101, construct a RepVGG-block, hereinafter referred to as block; the steps are as follows:

[0045] A multi-branch structure is constructed. The input P undergoes a 3x3 convolution and a BN normalization to obtain P1, with network parameters N1. Simultaneously, the image P undergoes a 1x1 convolution and a BN normalization to obtain P2, with network parameters N2. Finally, P undergoes a BN normalization to obtain the identity mapping P3, with network parameters N3, denoted as the Identity layer. A condition A is set in the Identity layer branch: if the RepVGG-block module does not change the number of feature channels and the convolution stride is set to 1, the block is a normal structure and remains unchanged; otherwise, this Identity layer branch is deleted, and downsampling is performed. The results of P1, P2, and P3 are summed. Then, the GELU function is used for activation. See the detailed structure below. Figure 3 -A, Figure 3 -B.

[0046] During the network training phase, the block structure remains unchanged. During the model deployment and inference phase, the network parameters for the two blocks in decision A are reparameterized, merging the multi-branch network structure into a single-path network structure. See details below. Figure 3 -C, Figure 3 -D.

[0047] The structure reparameterization operation is as follows:

[0048] The network parameters N2 from the 1x1 convolutional layer and BN layer, and N3 from the Identity layer are used for parameter inference and fused with the network parameters N1 from the 3x3 convolutional layer and BN layer. Simultaneously, the fused network structure is replaced with a 3x3 convolutional layer structure, and then the convolutional layers are stacked to form a single-path structure for inference and prediction.

[0049] For N2, a Conv2D convolutional layer is used to pad the 1x1 convolutional kernel into a 3x3 form. The result is added to N1, and then a 1x1 convolution is performed on the result to transform the 1x1 convolutional layer parameters into 3x3 convolutional layer network parameters.

[0050] For N3, a Conv2D convolutional layer is used, setting the kernel parameter of the current channel to 1 and the parameters of the remaining convolutional kernels to 0, padding it to a 3x3 convolutional form. The result is added to N1, and a 1x1 convolution is performed on the added result to transform it into the network layer parameters of a 3x3 convolutional layer.

[0051] S102. Construct a lightweight downsampling network structure using blocks. Stack these block modules into 8 layers. The number of input channels for each block is defined as the number of input feature channels, and the number of output channels is defined as [32, 32, 64, 64, 128, 128, 256, 256]. Layers 1, 3, 5, and 7 are downsampling layers; layers 2, 4, 6, and 8 represent the normal block structure. After activation by the GELU function in layer 1, a CBAM module is added; after activation by the GELU function in layer 8, a CBAM module is added, completing the network construction. Inputting an image into the network yields a feature map with 256 channels, defined as... See details. Figure 2 - Step 1.

[0052] The CBAM module, consisting of spatial attention and channel attention, transforms the original image into a multi-channel feature map during convolutional feature extraction and assigns weights to the features, making them easier for computers to recognize. The CBAM structure is shown below. Figure 4 .

[0053] Step S2: Extract feature points and calculate descriptors from the multi-channel feature map. See details. Figure 2 Step 2, the steps are as follows:

[0054] S201. Divide the feature F into two parts, F1 and F2. Activate F1 using the sigmoid function to obtain a predicted probability map, denoted as convf. For convf, perform upsampling using PixelShuffle to change the pixel size of the probability map, making convf a single-channel map and restoring it to the pixel size of the original input image, denoted as ConvF. Perform non-maximum suppression on ConvF, denoted as NMS-F, and determine the feature point coordinates of the original input image based on NMS-F, denoted as POS-F.

[0055] S202. Calculate the norm of the first channel of F2, divide F2 by this norm to obtain a low-resolution dense feature map, denoted as convd. Based on POS-F, perform bilinear sampling on the corresponding feature vectors in convd, followed by binarization. Extract the binary descriptor corresponding to each feature point, as shown in the following formula:

[0056]

[0057] Where z is the coordinate of the feature map, and f is the compressed-scale feature map extracted from convd.

[0058] Combined with binarized descriptors The final feature points are extracted using the feature point coordinates POS-F.

[0059] S203: Configure the AdamW optimizer to optimize the binary descriptor, which is dominated by 0s and 1s. The cross-loss entropy function is calculated with an initial learning rate of 0.0001 and a maximum number of iterations of 100 epochs. The learning rate is multiplied by a coefficient of 0.55 every 25 epochs.

[0060] Step S3: Perform feature matching using the FLANN algorithm. The specific steps are as follows:

[0061] The POS-F obtained in S201 and the corresponding [information] in S202 Save the data as a dictionary and use the FLANN algorithm to perform feature matching.

[0062] Step S4: Use the improved RANSAC algorithm to remove mismatches and obtain the final matching pairs. The specific steps are as follows:

[0063] The improvement is to denote the set of matching point pairs obtained in step S3 as... The matched pairs are arranged in descending order of quality. The first 60% of the pairs are selected and denoted as set M. When set M contains fewer than 100 elements, the set is... Let M be the set. Then, the RANSAC algorithm is used to randomly select 4 point pairs from the set M to calculate the homography transformation matrix H, and the interior points are filtered to remove mismatches.

[0064] The feature point quality evaluation index is the NMS-F value of the predicted probability map in S2, and the matching point pair quality is the sum of the NMS-F values ​​of the feature point pairs in the matching image.

[0065] The above embodiments are merely preferred technical solutions of the present invention and should not be considered as limitations on the present invention. The scope of protection of the present invention should be limited to the technical solutions described in the claims, including equivalent substitutions of the technical features described in the claims. That is, equivalent substitutions and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. An image matching method based on the RepVGG lightweight network, characterized in that... The following steps are involved: Step S1: Construct a lightweight feature extraction network to extract multi-channel feature maps; Step S2: Extract feature points and calculate descriptors from the multi-channel feature map; Step S3: Perform feature matching using the FLANN algorithm; Step S4: Use the improved RANSAC algorithm to remove mismatches and obtain the final matching pairs; In step S1, a downsampling network with RepVGG as the backbone is constructed, and feature maps are sampled using the following method: S101. Construct a RepVGG-block: Construct a multi-branch structure. Input P undergoes a 3x3 convolution and a BN normalization to obtain P1, with network parameters N1; simultaneously, image P undergoes a 1x1 convolution and a BN normalization to obtain P2, with network parameters N2; furthermore, P undergoes a BN normalization to obtain the identity mapping P3, with network parameters N3, denoted as the Identity layer; in the Identity layer branch, set the judgment A. Judgment A: Only when the RepVGG-block module does not change the number of feature channels and the convolution stride is set to 1, is the RepVGG-block a normal structure and remains unchanged; otherwise, delete the Identity layer branch here, and then perform downsampling; add the results of P1, P2, and P3; then activate using the GELU function. When the network is in the training phase, the above RepVGG-block structure remains unchanged; when the network is in the deployment model inference phase, the network parameters of the two RepVGG-blocks in judgment A are reparameterized, and the multi-branch network structure is merged into a single-path network structure. S102. Construct a lightweight downsampling network structure using RepVGG-block. Stack the RepVGG-block module into 8 layers. The number of input channels of the RepVGG-block is defined as the number of input feature channels, and the number of output channels is defined as [32, 32, 64, 64, 128, 128, 256, 256]. Among them, layers 1, 3, 5, and 7 are downsampling layers; layers 2, 4, 6, and 8 are normal RepVGG-block structures. After activation by the GELU function in layer 1, a CBAM module is added; after activation by the GELU function in layer 8, a CBAM module is added, and the network construction is completed. Input the image into the network to obtain a feature map with 256 channels, defined as... ; The operation of step S4 is as follows: Let the set of matching point pairs obtained in step S3 be denoted as The matched pairs are arranged in descending order of quality. The first 60% of the pairs are selected and denoted as set M. When set M contains fewer than 100 elements, the set is... Let M be the set of points. Then, the RANSAC algorithm is used to randomly select 4 point pairs from set M to calculate the homography transformation matrix H, and the inliers are filtered to remove mismatches. The feature point quality evaluation index is the NMS-F value of the predicted probability map in S2, and the matching point pair quality is the sum of the NMS-F values ​​of the feature point pairs in the matching image.

2. The image matching method based on the RepVGG lightweight network according to claim 1, characterized in that: The decomposition steps of step S2 are as follows: S201. Divide the feature F into two parts, F1 and F2. Activate F1 with the sigmoid function to obtain the predicted probability map, denoted as convf. Upsample convf using PixelShuffle to change the pixel size of the probability map, making convf a single channel and restoring it to the original input image pixel size, denoted as ConvF. Non-maximum suppression is applied to ConvF, denoted as NMS-F. The feature point coordinates of the original input image are determined based on NMS-F, denoted as POS-F. S202. Calculate the norm of the first channel of F2, divide F2 by this norm, and obtain the low-resolution dense feature map, denoted as convd; Based on POS-F, bilinear sampling is performed on the corresponding feature vectors in convd, followed by binarization. Extract the binary descriptor corresponding to each feature point, as shown in the following formula: ； Where z is the coordinate of the feature map, and f is the compressed-scale feature map extracted from convd; Combined with binarized descriptors Using the feature point coordinates POS-F, extract the final feature points; S203: Configure the AdamW optimizer to optimize the binary descriptor, which is dominated by 0s and 1s. The cross-loss entropy function is calculated with an initial learning rate of 0.0001 and a maximum number of iterations of 100 epochs. The learning rate is multiplied by a coefficient of 0.55 every 25 epochs.

3. The image matching method based on the RepVGG lightweight network according to claim 1, characterized in that: In step S3, when performing feature matching using the FLANN algorithm, the POS-F obtained in step S201 and the corresponding features in S202 are used... Save the data as a dictionary and use the FLANN algorithm to perform feature matching.

4. The image matching method based on the RepVGG lightweight network according to claim 1, characterized in that: The structure reparameterization operation is as follows: The network parameters N2 of the 1x1 convolutional layer and BN layer, and the network parameters N3 of the Identity layer are used for parameter inference and fused with the network parameters N1 of the 3x3 convolutional layer and BN layer. At the same time, the fused network structure is replaced with the network structure of the 3x3 convolutional layer, and then the convolutional layers are stacked to form a single-path structure for inference and prediction. For N2, the 1x1 convolutional kernel is padded to a 3x3 form through the Conv2D convolutional layer; the result is added to N1, and the result after addition is convolved with 1x1, that is, the parameters of the 1x1 convolutional layer are transformed into the parameters of the 3x3 convolutional layer network. For N3, the Conv2D convolutional layer sets the kernel parameter of the current channel to 1 and the kernel parameters of the rest to 0, padding it to a 3x3 convolutional form. The result is added to N1, and a 1x1 convolution is performed on the result to transform it into the network layer parameters of a 3x3 convolutional layer.

5. The image matching method based on the RepVGG lightweight network according to claim 1, characterized in that: In step S1, the CBAM module is connected by spatial attention and channel attention. During the convolution feature extraction process, the original image is transformed into a multi-channel feature map, and weights are assigned to the features, making them easier for the computer to recognize.

Images