Multi-angle feature point matching method based on contour extraction

Abstract

The application relates to the technical field of image processing, and discloses a multi-angle feature point matching method based on contour extraction, which comprises the following steps: inputting an original image into a contour extraction module to obtain an edge weight graph; inputting the original image into a descriptor head to obtain a confidence matrix and a feature descriptor; adaptively weighting the confidence matrix by using the edge weight graph to obtain a weighted confidence matrix; inputting the original image into a key point head to obtain a key point heat map; screening the first L key points in the key point heat maps corresponding to the original images taken at two angles to perform matching; if the matching is successful, splicing the descriptors corresponding to the matched key points in the feature descriptor to perform multilayer perception machine processing to obtain a shift probability distribution; and searching for the positions of the key points in the original image based on the shift probability distribution, so that feature point matching is completed. The application aims to improve the accuracy of feature point matching and reduce the false matching rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-angle feature point matching method based on contour extraction. BACKGROUND

[0002] When a target image is photographed from multiple angles, the feature points in the images of each angle need to be matched. In the existing feature point matching method, the feature points are often too concentrated, such as a large number of feature points gathering in an image region with rich texture, and the feature points are extremely rare in a region with single texture. Such uneven distribution of feature points leads to a large mismatch during matching. Therefore, in the application of multi-angle photographing and feature point matching, a method is needed to select reasonable feature points and improve the accuracy of feature point matching. SUMMARY

[0003] The purpose of the present application is to improve the accuracy of feature point matching and reduce the mismatch rate, and to provide a multi-angle feature point matching method based on contour extraction.

[0004] In order to achieve the above-mentioned purpose of the application, the embodiments of the present application provide the following technical solutions:

[0005] The multi-angle feature point matching method based on contour extraction is realized based on a feature point matching model, and the feature point matching model includes a contour extraction module and a feature point extraction module. The method includes the following steps:

[0006] Step 1: input the original image into the contour extraction module to obtain an edge weight map;

[0007] Step 2: the feature point extraction module includes a descriptor head, a key point head, and a dense matching unit; input the original image into the descriptor head to obtain a confidence matrix and a feature descriptor; use the edge weight map to adaptively weight the confidence matrix to obtain a weighted confidence matrix; input the original image into the key point head to obtain a key point heat map;

[0008] Step 3: based on the weighted confidence matrix, the first L key points in the key point heat maps corresponding to the original images photographed from two angles are matched, if the matching is successful, the descriptors corresponding to the matched key points in the feature descriptors are spliced and then processed by a multi-layer perception machine to obtain a displacement probability distribution; based on the displacement probability distribution, the positions of the key points in the original images are found, thereby completing the feature point matching.

[0009] Compared with the prior art, the present application has the beneficial effects that: the present application uses a feature point matching mode based on edge extraction, focuses on the image edge, and can effectively disperse the distribution of feature points. According to the trend of the edge and the feature, the feature points are reasonably selected to optimize the distribution of the feature points and enhance the matching accuracy, thereby solving the problem of false matching caused by the concentration of feature points in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0011] Figure 1 The method flowchart of the present application is shown in the figure.

[0012] Figure 2 The network structure schematic diagram of the profile extraction module of the embodiment of the present application is shown in the figure.

[0013] Figure 3 The network structure schematic diagram of the description sub-head of the embodiment of the present application is shown in the figure.

[0014] Figure 4 The network structure schematic diagram of the key point head of the embodiment of the present application is shown in the figure.

[0015] Figure 5 The schematic diagram of the embodiment of the present application using the edge weight map to adaptively weight the confidence matrix is shown in the figure.

[0016] Figure 6 The flowchart of the dense matching unit of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0017] The technical solutions of the embodiments of the present application will be described in detail below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0018] It should be noted that similar reference numerals and letters refer to like items in the accompanying drawings and as such once an item is defined in one drawing, further definitions and explanations within subsequent drawings are omitted. Also, in the description of the present application, the terms "first", "second", and the like are used to distinguish between similar items, and are not used to indicate or imply relative importance or a particular order of or sequence between the entities or operations. Additionally, the terms "connected", "coupled", and the like, refer to any connection or coupling between two elements, which connection or coupling can be direct or indirect, and can be physical, electrical, or both.

[0019] Embodiment:

[0020] The present application is realized by the following technical solutions, as shown in a multi-angle feature point matching method based on contour extraction, which is realized based on a feature point matching model, the feature point matching model comprising a contour extraction module and a feature point extraction module, the method comprising the following steps: Figure 1

[0021] Step 1, input the original image into the contour extraction module to obtain an edge weight map.

[0022] The original image is input into the contour extraction module, please refer to Figure 2 , the contour extraction module comprising a backbone network, a USNet network, and a double fusion (dfuse) unit.

[0023] The backbone network comprises three Block layers, namely a first Block layer, a second Block layer, and two third Block layers; each Block layer comprises two CNN layers (i.e. Conv1 and Conv2), a BN layer, and two smish activation functions (i.e. smish1 and smish2), and the internal connection relationship is [Conv1+BN+smish1+Conv2+smish2].

[0024] The USNet network comprises a first USNet unit, a second USNet unit, and a third USNet unit; the first USNet unit and the second USNet unit each comprise a convolution layer (Conv, kernel size 1x1), a smish activation function, and a deconvolution layer (DEConv, kernel size 2x2) connected in sequence; and the third USNet unit comprises a convolution layer (Conv, kernel size 1x1), a convolution layer (Conv, kernel size 4x4), a convolution layer (Conv, kernel size 1x1), a smish activation function, and a deconvolution layer (DEConv, kernel size 4x4) connected in sequence.

[0025] The double fusion unit comprises a concatenation layer (Concat), a first separable convolution layer (DWConv1), and a second separable convolution layer (DWConv2).​

[0026] Based on the above network structure, the working principle of the contour extraction module is as follows:

[0027] First, the original image F0 is input into the first Block layer to obtain the feature map F1; the feature map F1 is input into the second Block layer to obtain the feature map F2; the feature map F1 and the feature map F2 are jointly input into the first third Block layer to obtain the feature map F3; and the feature F2 and the feature map F3 are jointly input into the second third Block layer to obtain the feature map F4.

[0028] Next, the feature map F1 is input into the first USNet unit for edge feature extraction to generate an edge prediction map with the same size as the input The feature map F2 is input into the second USNet unit for edge feature extraction to generate an edge prediction map with the same size as the input The feature map F2 and the feature map F4 are jointly input into the third USNet unit for edge feature extraction to generate an edge prediction map with the same size as the input The edge prediction map The edge prediction map And the edge prediction map are spliced through the splicing layer to generate the edge prediction map The related formula is as follows:

[0029]

[0030] Wherein, σ represents the ReLU activation function; USNet represents the processing of the USNet unit; h i represents the input of the i-th USNet unit; and Concat represents the splicing operation.

[0031] Then, the edge prediction map is input into the first separable convolution layer to obtain the feature edge map The feature edge map is input into the second separable convolution layer to obtain the feature edge map The feature edge map and the feature edge map are fused for the first time; and the activation function and the pixel-by-pixel increase method are used to fuse the result of the first fusion for the second time, to perform nonlinear activation on the feature edge map, and to perform normalization processing and weight mapping, so as to obtain the edge weight map The related formula is as follows:

[0032]

[0033] Wherein, h dfusedenotes the result of the first fusion; ewa denotes a pixel-wise operation; denotes the smish activation function; and denotes the ReLU activation function.

[0034] In step 2, the feature point extraction module comprises a descriptor head, a key point head, and a dense matching unit; the original image is input into the descriptor head to obtain a confidence matrix and a feature descriptor; the confidence matrix is adaptively weighted using an edge weight map to obtain a weighted confidence matrix; the original image is input into the key point head to obtain a key point heat map.

[0035] See Figure 3 , the descriptor head comprises six convolution blocks and a fusion block; the six convolution blocks are respectively a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block, and a sixth convolution block; each convolution block comprises a 2D convolution layer (kernel size is 1 or 3), a ReLU activation function, a BatchNorm normalization layer, and a basic layer (step size is 2). An original image (processed as a grayscale image) with a size of HxW and a channel of C (C=1) is input into the first convolution block to obtain a feature map P1 with a scale of HxWx4; the feature map P1 is input into the second convolution block to obtain a feature map P2 with a scale of H / 2xW / 2x8; the feature map P2 is input into the third convolution block to obtain a feature map P3 with a scale of H / 4xW / 4x24; the feature map P3 is input into the fourth convolution block to obtain a feature map P4 with a scale of H / 8xW / 8x64; the feature map P4 is input into the fifth convolution block to obtain a feature map P5 with a scale of H / 16xW / 16x64; the feature map P5 is input into the sixth convolution block to obtain a feature map P6 with a scale of H / 32xW / 32x128. The feature maps P4, P5, and P6 with scales of 1 / 8, 1 / 16, and 1 / 32 are upsampled, projected, and element-wise summed to a scale of H / 8xW / 8x64, and then fused by element-wise addition to obtain a feature map G0 with a scale of H / 8xW / 8x64. The feature map G0 is input into the fusion block for further multi-scale information integration to generate a feature descriptor G.

[0036] In addition, see Figure 3 , a convolution block is further introduced in the descriptor head to encode the feature map G0 to obtain a confidence matrix D H / 8×H / 8×1 , each local feature G i,j in the feature descriptor G i,j .

[0037] See Figure 5 , the edge weight map is spatially aligned with the scale of the confidence matrix D to obtain a spatially aligned edge weight map Then the aligned edge weight map The confidence matrix D is adaptively weighted so that the weighted confidence matrix D' has the performance of edge enhancement. Each local feature G in the feature descriptor G is matched with the weighted confidence D' i,j i,j .

[0038] Referring to Figure 4 , the original image is reduced to 1 / 8 resolution and divided into multiple grids, each grid is an 8x8 dimensional feature vector (so the number of grids is affected by the resolution of the original image), and a grid is input into the key point head, and a key point heat map K e R H / 8×H / 8×65 is generated after processing by a 1x1 convolutional layer, and the grid corresponds to 64 key points k i,j e K in the key point heat map.

[0039] Step 3, based on the weighted confidence matrix, the first L key points in the key point heat map corresponding to the original image of the two angle shots are matched, if the matching is successful, the matching successful key points are concatenated in the corresponding descriptor in the feature descriptor, and then the multi-layer perception is processed to obtain the offset probability distribution; based on the offset probability distribution, the position of the key point in the original image is found, thereby completing the feature point matching.

[0040] In the above manner, referring to Figure 6 , the same position grid of two original images of different angles can obtain corresponding key point heat maps K1 and K2. The key point heat map, the confidence matrix, and the feature descriptor are input into the dense matching unit, the first L key points with the highest confidence value in the key point heat map K1 are selected, the first L key points with the highest confidence value in the key point heat map K2 are selected, and the L key points in K1 are matched with the L key points in K2, if the matching is successful, the matching successful key point pair is concatenated in the descriptor in the feature descriptor G and input into the multi-layer perception for processing to obtain the offset probability distribution. For example, the la-th key point in K1 is matched with the lb-th key point in K2, then the descriptor ga corresponding to the la-th key point in the feature descriptor G corresponding to the first original image is obtained, and the descriptor gb corresponding to the lb-th key point in the feature descriptor G corresponding to the second original image is obtained, the descriptor ga and the descriptor gb are concatenated and input into the multi-layer perception (MLP) as a descriptor pair with a scale of 1x1x128 to obtain an offset probability distribution o, and the scale of the offset probability distribution is 1x1x64.

[0041] ​Since the key point heat map K is obtained at 1 / 8 resolution of the original image, after obtaining the offset probability distribution o, a position coordinate is determined by returning to the grid where the offset probability distribution o is located, and the feature point coordinate at the original image resolution is restored through the position coordinate and the position coordinate of the key point, so that the feature point matching of the two original images can be realized.

[0042] In the process of acquiring the edge weight map in the contour extraction module, adaptive morphological opening operation is performed, and a 2*3 elliptical kernel is used to remove small edges, and the mathematical expression is:

[0043]

[0044] Among them, E open represents the image after opening operation; E represents the original image; B represents the dynamic kernel size; represents the erosion operation; and represents the dilation operation.

[0045] The process of normalizing and weight mapping the feature edge map is: Gamma correction is used for nonlinear normalization. Gamma correction belongs to power transformation in nature, which maps the gray value of the pixel point in the feature edge map, and the mathematical expression is:

[0046] s = cr γ (r∈[0,1],γ>0)

[0047] Among them, s represents the mapped edge response intensity value; c and γ represent the coefficients; and r represents the gray value before mapping.

[0048] When 0<γ<1, the function curve has a larger slope in the low brightness area such as weak edge, so that the gray difference of weak edge is nonlinearly amplified; in the high brightness area such as strong edge or high light over-response, the curve slope is reduced, and the excessive diffusion of pixel value is suppressed. In this module, the normalization formula of the target contour image is as follows:

[0049]

[0050] Among them, E norm represents the image after opening operation; E open represents the original image; E max represents the normalized maximum value; and E min represents the normalized minimum value. The exponential term is set to 0.8 in order to retain more weak edge information while suppressing the over-response of the high light area. The finally output edge weight map not only represents the edge confidence, but also reflects the structural significance degree of the edge, which provides guiding information for subsequent key point screening.

[0051] In the process of using the edge weight map​ In the process of adaptive weighting of the confidence matrix D, the edge region response is enhanced by exponential weighting:

[0052]

[0053] wherein D` i,j represents the confidence after adaptive weighting; and a represents an edge enhancement coefficient. represents the edge weight map after spatial alignment of the confidence matrix D.

[0054] In the process of screening the first L key points, a double-threshold screening mechanism is adopted to filter the low-response region while retaining the weak response of the edge region. The double-threshold screening mechanism is:

[0055]

[0056] wherein D`` i,j represents the confidence of the first L key points screened out; τ1 represents a global threshold for screening a high-response region; and τ2 represents an edge region threshold for retaining a key point with weak response but belonging to an edge region.

[0057] A binary classification weighted cross-entropy loss function is used to measure the performance of the contour extraction module:

[0058] L wce =-w1×y×log(y`) -w0×(1-y)×log(1-y`)

[0059] wherein L wce represents the loss function of the contour extraction module; w1 and w0 are weight coefficients; y is a label real value between 0 and 1, indicating whether a certain pixel point is an edge point; and y` is a prediction value of whether a certain pixel point is an edge point.

[0060] The loss function of the descriptor head is:

[0061]

[0062] wherein L ds represents the loss function of the descriptor head; N represents the number of matched key point pairs; s i,i represents the similarity between the descriptor i corresponding to the first image and the descriptor i corresponding to the second image; and s i,j represents the similarity between the descriptor i corresponding to the first image and the descriptor j corresponding to the second image.

[0063] The key point positioning accuracy loss function is:

[0064]

[0065] wherein, L IoU represents a key point positioning accuracy loss function; p i,j represents a key point in the predicted key point heat map; k i,j represents a key point in the real key point heat map.

[0066] The edge distribution alignment loss function is:

[0067]

[0068] wherein, E i,j represents the gray value of the point with coordinates (i, j) in the edge ground truth map; H and W are the height and width of the edge ground truth map respectively; ∈ = 10 -6 represents a small quantity for numerical stability.

[0069] The key point head loss function is:

[0070] L KeyPoint = β1L IoU + β2L Edge

[0071] wherein, β1 and β2 are weight coefficients.

[0072] The total loss function is:

[0073] L total = λ1L KeyPoint + λ2L ds + λ3L wce

[0074] wherein, λ1, λ2 and λ3 are weight coefficients.

[0075] To sum up, the present application solves the problem of large error when the feature points are too dense for the dense matching task in the feature point matching task, and solves the problem that some feature points of interest are not selected or filtered out because the selected feature points are too concentrated for the feature point matching in the image registration task.

[0076] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multi-angle feature point matching method based on contour extraction, characterized in that: Based on the feature point matching model, the feature point matching model includes a contour extraction module, a feature point extraction module; The method comprises the following steps: Step 1, input the original image into the contour extraction module, and obtain the edge weight map; In step 1, the contour extraction module includes a backbone network, a USNet network and a double fusion unit; The backbone network includes three Block layers, namely a first Block layer, a second Block layer and two third Block layers; The USNet network includes a first USNet unit, a second USNet unit and a third USNet unit; The double fusion unit includes a splicing layer, a first separable convolution layer and a second separable convolution layer; The original image F0 is input into the first Block layer to obtain the feature map F1; The feature map F1 is input into the second Block layer to obtain the feature map F2; The feature map F1 and the feature map F2 are input into the first third Block layer to obtain the feature map F3; The feature F2 and the feature map F3 are input into the second third Block layer to obtain the feature map F4; The feature map F1 is input into the first USNet unit for edge feature extraction to generate an edge prediction map with the same size as the input The feature map F2 is input into the second USNet unit for edge feature extraction to generate an edge prediction map with the same size as the input The feature map F2 and the feature map F4 are input into the third USNet unit for edge feature extraction to generate an edge prediction map with the same size as the input The edge prediction map The edge prediction map The edge prediction map The edge prediction map is generated by splicing through a splicing layer The relevant formula is: wherein, σ denotes a ReLU activation function; USNet denotes the processing of a USNet unit; h i denotes the input of the i-th USNet unit; Concat denotes a concatenation operation; The edge prediction map Input a first separable convolution layer to obtain a feature edge map The feature edge map Input a second separable convolution layer to obtain a feature edge map The feature edge map The feature edge map Firstly, the feature edge maps are fused; secondly, the first fusion result is fused again by using an activation function and a pixel-by-pixel increase method, the feature edge maps are activated in a nonlinear manner, and normalization processing and weight mapping are performed, so as to obtain an edge weight map The relevant formula is: where h dfuse denotes the result of the first fusion; ewa denotes the pixel-wise operation; denotes the smish activation function; denotes the ReLU activation function; Step 2, the feature point extraction module includes a descriptor head, a key point head and a dense matching unit; The original image is input into the descriptor head to obtain a confidence matrix and a feature descriptor; The confidence matrix is adaptively weighted using the edge weight map to obtain a weighted confidence matrix; The original image is input into the key point head to obtain a key point heat map; Step 3, based on the weighted confidence matrix, the first L key points in the key point heat map corresponding to the original image pair shot from two angles are matched, if the matching is successful, the matching successful key points are spliced in the corresponding descriptor in the feature descriptor, and then a multilayer perception machine is used for processing to obtain a displacement probability distribution; The position of the key point in the original image is found based on the displacement probability distribution, so as to complete the feature point matching. 2.The multi-angle feature point matching method based on contour extraction according to claim 1, characterized in that: In step 2, the step of inputting the original image into the descriptor head to obtain the confidence matrix and the feature descriptor comprises: The descriptor head includes six convolution blocks and a fusion block; The six convolution blocks are a first convolution block, a second convolution block, a third convolution block, a fourth convolution block, a fifth convolution block and a sixth convolution block; The first convolutional block is input with a raw image with a size of HxW and a channel of C, C=1, to obtain a feature map P1 with a scale of HxWx4; the feature map P1 is input into the second convolutional block to obtain a feature map P2 with a scale of H / 2xW / 2x8; the feature map P2 is input into the third convolutional block to obtain a feature map P3 with a scale of H / 4xW / 4x24; the feature map P3 is input into the fourth convolutional block to obtain a feature map P4 with a scale of H / 8xW / 8x64; the feature map P4 is input into the fifth convolutional block to obtain a feature map P5 with a scale of H / 16xW / 16x64; the feature map P5 is input into the sixth convolutional block to obtain a feature map P6 with a scale of H / 32xW / 32x128; the feature map P4, the feature map P5 and the feature map P6 are up-sampled, projected and summed element by element to the scale of H / 8xW / 8x64, and then fused by element by element addition to obtain a feature map G0 with a scale of H / 8xW / 8x64; the feature map G0 is input into the fusion block for further multi-scale information integration to generate a feature descriptor G; By describing the convolution block in the descriptor sub-head, the feature encoding is performed on the feature map G0, and the confidence matrix D ∈ R H / 8×H / 8×1, each local feature G in the feature descriptor G i,j There is a matching confidence D in the confidence matrix D i,j . 3.The multi-angle feature point matching method based on contour extraction according to claim 2, characterized in that: In the step 2, the step of using the edge weight map to adaptively weight the confidence matrix to obtain a weighted confidence matrix comprises: aligning the scale of the edge weight map with the scale of the confidence matrix D to obtain a spatially aligned edge weight map Then using the aligned edge weight map to adaptively weight the confidence matrix D, so that the weighted confidence matrix D` has the performance of edge enhancement; each local feature G in the feature descriptor G i,j is matched with the weighted confidence D` i,j .

4. The multi-angle feature point matching method based on contour extraction according to claim 3, characterized in that: In the step 2, the step of inputting the raw image into the key point head to obtain the key point heat map comprises: After the original image is reduced to 1 / 8 resolution and divided into multiple grids, each grid is an 8x8-dimensional feature vector, and a grid is input into the key point head respectively, and a key point heat map K is generated after processing by a 1x1 convolution layer H / 8×H / 8×65 The grid corresponds to a key point heat map containing 64 key points k i,j ∈K.

5. The method of claim 4, wherein the method further comprises: The step 3 specifically comprises the following steps: The same position grid of two raw images with different angles respectively obtains corresponding key point heat maps K1 and K2; the respective key point heat map, the confidence matrix and the feature descriptor are input into the dense matching unit, the first L key points with the highest confidence value in the key point heat map K1 are selected, the first L key points with the highest confidence value in the key point heat map K2 are selected, and the L key points in the key point heat map K1 are matched with the L key points in the key point heat map K2 respectively; If the la-th key point in the key point heat map K1 and the lb-th key point in the key point heat map K2 are successfully matched, then the descriptor ga corresponding to the la-th key point in the feature descriptor G1 and the descriptor gb corresponding to the lb-th key point in the feature descriptor G2 are spliced and input into the multi-layer perception machine for processing to obtain an offset probability distribution o; A position coordinate is determined by returning to the grid through the offset probability distribution o, and the feature point coordinate under the resolution of the original image is restored through the position coordinate and the position coordinate of the key point, so that the feature point matching of the two raw images can be realized.

Images