Heterogeneous image matching method based on twin neural network

Abstract

The application relates to a heterogeneous image matching method based on a twin neural network, and comprises the following steps: acquiring a plurality of template images and corresponding target images; inputting the template images and the corresponding target images into a twin neural network to generate a feature point heat map branch; determining the positions of feature points by using the feature point heat map branch, and matching the feature points in the target images to the feature points in the template images; the twin neural network generates a feature vector branch; and according to the matched feature points, the feature vectors of the feature points are extracted by using the feature vector branch. The application solves the feature mapping problem of heterogeneous images by using high-latitude semantic information extracted by the twin neural network, and realizes the matching of the feature points on the basis, so as to solve the matching problem between the heterogeneous images.

Description

Technical Field

[0001] This invention relates to the field of image feature matching technology, and in particular to a heterogeneous image matching method based on Siamese neural networks. Background Technology

[0002] Image matching technology based on digital image processing utilizes information such as edge textures in images to solve image matching and registration problems. However, this solution is only suitable for matching images from the same source (such as matching visible light images with visible light images, infrared images with infrared images, SAR images with SAR images, etc.), and cannot solve the task of matching and registering images from different sources. This is because the differences in the shooting equipment and imaging principles of images from different sources result in significant differences in the appearance of the resulting images, making it impossible to accurately match the same target area in different modalities.

[0003] Therefore, in heterogeneous image matching tasks, in addition to studying high-precision feature point detection and extraction, it is also necessary to study heterogeneous feature space mapping that achieves consistency. Summary of the Invention

[0004] The purpose of this invention is to solve the feature mapping problem of heterogeneous images by using high-dimensional semantic information extracted by Siamese neural networks, and to achieve feature point matching based on this information, thereby solving the matching problem between heterogeneous images and providing a heterogeneous image matching method based on Siamese neural networks.

[0005] To achieve the above-mentioned objectives, the embodiments of the present invention provide the following technical solutions:

[0006] The heterogeneous image matching method based on Siamese neural networks includes the following steps:

[0007] Step 1: Obtain several template images and their corresponding target images;

[0008] Step 2: Input the template image and the corresponding target image into the Siamese neural network to generate a feature point heatmap branch; use the feature point heatmap branch to determine the location of the feature points, and match the feature points in the target image to the feature points in the template image;

[0009] Step 3: The Siamese neural network generates feature vector branches; based on the matched feature points, the feature vector branches are used to extract the feature vectors of the feature points.

[0010] Furthermore, step 1 specifically includes the following steps: taking the entire original image as the target image and a partial crop of the original image as the template image, wherein the resolution of the target image and the template image are equal or different; performing data augmentation on the template image and the target image by means of color, angle, and contrast; and using the shared region of the template image and the target image after augmentation as the input of the Siamese neural network.

[0011] Furthermore, step 2, the step of determining the location of feature points using feature point heatmap branches, includes:

[0012] In a Siamese neural network, the encoder and decoder generate a heatmap. The heatmap is then subjected to max pooling with a fixed step size. The pooling results are compared with the heatmap, and points with the same values ​​are selected as candidate feature points.

[0013] Set a confidence threshold, and select candidate feature points whose values ​​are greater than the confidence threshold as feature points output by the feature point heatmap branch, and output the location of the feature points at the same time.

[0014] Furthermore, step 2, the step of matching feature points in the target image to feature points in the template image, includes:

[0015] The feature points output from the feature point heatmap branch are used as positive samples, and feature points are randomly selected from the non-shared regions of the template image and the target image as negative samples. Cosine similarity is used to measure the feature similarity of each feature point.

[0016] A global search method is used to calculate the association between feature points in the target sample and feature points in the template sample one by one, in order to find the globally optimal matching relationship:

[0017]

[0018] Among them, P i,j H(P) represents the element value that maps from the i-th row of the target sample to the j-th row of the template sample; H(P) is the value of P. i,j Entropy;

[0019]

[0020] Where P represents a mapping where all rows of the target sample sum to vector a and all columns of the template sample sum to vector b, and this mapping is U(a,b); C represents the cost of completing the element mapping of P; ∈ represents the regularization coefficient;

[0021] P = u i K i,j v j u i With v ju represents a set of solutions obtained from the mapping. i Let v be the displacement of the feature point in the i-th row of the target image. j This represents the displacement of the feature point in the j-th row of the template image;

[0022]

[0023] By u i v j The relationship between the feature points in the template image and the feature points in the target image is solved to obtain the optimal connection relationship.

[0024] Furthermore, in step 2, after matching feature points in the target image to feature points in the template image, a step of verifying the accuracy of the matching result is also included:

[0025] Let PT be the feature points within the target image. target Feature points PT within the template image template If a matching is achieved, then there exist matrices A and B such that the feature point PT... template The coordinates of the feature points PT are transformed through matrix operations. target :

[0026] PT target =PT template ×A+B

[0027] Where matrix A is a 2×2 matrix and matrix B is a 2×1 matrix:

[0028]

[0029]

[0030] Let feature point PT target The coordinates are x1, y1, and the feature point is PT. template If the coordinates of the two feature points are x2 and y2, then the mapping relationship between the two feature points is:

[0031]

[0032]

[0033]

[0034]

[0035] x1 = a1x2 + b1y2 + c1

[0036] y1 = a²x² + b²y² + c²

[0037] Define matrices A and B as mapping matrices M for feature points, and then map the feature points PT... template With feature point PT target Solve the problem to verify the accuracy of the matching results.

[0038] Furthermore, step 2 also includes the step of using regression branches to predict the bias of the position coordinates (x, y) of the feature points, with a bias coefficient of 0 to 1, thereby improving the accuracy of the feature point positions.

[0039] Furthermore, in step 3, the step of extracting the feature vector of the feature point using the feature vector branch based on the matched feature point includes: extracting features from the encoder and decoder, performing feature interpolation on the feature points, and thus generating the feature vector.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] This invention enables matching or registration between heterogeneous images, ensuring the detection of directly related feature points at the target object's location and evaluating consistent feature points between heterogeneous images. After feature point detection, feature vectors are used to calculate the similarity between the two sets of matched feature points. The quality of feature description directly affects the accuracy of image registration. While ensuring the accuracy of same-source matching, this invention also addresses the impact of appearance differences between heterogeneous images on feature vector extraction. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the twin neural network structure according to an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of a twin neural network structure with a regression branch added in an embodiment of the present invention;

[0045] Figure 3 This is a diagram showing the matching effect between feature points in the target image and feature points in the template image in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0047] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance, or suggesting any such actual relationship or order between these entities or operations. Additionally, the terms "connected," "linked," etc., can refer to a direct connection between elements or an indirect connection via other elements.

[0048] Example:

[0049] This invention is achieved through the following technical solution: a heterogeneous image matching method based on Siamese neural networks, comprising the following steps:

[0050] Step 1: Obtain several template images and their corresponding target images.

[0051] One approach is to use the entire original image as the target image and a local crop of the original image as the template image, in which case the target image and the template image are from the same source. Data augmentation is performed on the template image and the target image by adjusting color, angle, and contrast. After augmentation, the shared region between the template image and the target image is used as the input to the Siamese neural network for training.

[0052] As another approach, a target image is acquired, and an image from a different source than the target image is used as a template image. In this case, the target image and the template image are different source images; data augmentation is performed in the same way.

[0053] The resolutions of the target image and the template image may be equal or unequal.

[0054] Step 2: Input the template image and the corresponding target image into the Siamese neural network to generate a feature point heatmap branch; use the feature point heatmap branch to determine the location of the feature points, and match the feature points in the target image to the feature points in the template image.

[0055] Please see Figure 1 The Siamese neural network uses the Hourglass structure to encode and decode image information, generating two branches: a feature point heatmap branch and a feature vector branch. The feature point heatmap branch is used to determine the location of feature points and match feature points in the target image to feature points in the template image. The feature vector branch is used in step 3 to extract feature vectors of feature points based on the matched feature points.

[0056] The feature point heatmap branch outputs the set of highlighted points in the heatmap as feature points. The heatmap undergoes max pooling with a fixed step size (kernel_size = 7 or 15), and the pooling result is compared with the original heatmap. Points with the same value in the heatmap before and after pooling are defined as candidate feature points. A confidence threshold is introduced to control the total number of feature points that can be detected in the image. Feature point detection is performed on the shared region after augmentation of the template and target images. The union of the detected feature points is used as the ground truth to supervise the regression of the feature point heatmap.

[0057] bool_mask=heat==max_pool(heat)

[0058] conf_mask = heat > thresh

[0059] keypoints_mask=bool_mask&conf_mask

[0060] Where heat is the heatmap, max_poo(heat) is the heatmap after max pooling, bool_mask represents the coordinates of the same value in the heatmap before and after pooling, and conf_mask represents the coordinates of the heatmap where the value is greater than the confidence threshold thresh. The keypoint_mask is obtained by taking the union of bool_mask and conf_mask as the final detection result of the feature point.

[0061] To ensure the correct matching relationship between feature points in the target image and feature points in the template image, a global search method is used to calculate the association relationship between feature points in the target image and feature points in the template image one by one, in order to find the globally optimal matching relationship:

[0062]

[0063] Among them, P i,j H(P) represents the element value that maps from the i-th row of the target image to the j-th row of the template image; H(P) is the value of P. i,j Entropy;

[0064]

[0065] in, The regularization objective is represented by U(a,b); P represents a mapping where all rows of the target sample are summed to form vector a and all columns of the template sample are summed to form vector b; C represents the cost of completing the element mapping of P; ∈ represents the regularization coefficient.

[0066] P = u i K i,j v j u i With v j u represents a set of solutions obtained from the mapping. i Let v be the displacement of the feature point in the i-th row of the target image. j K represents the displacement of the feature point in the j-th row of the template image. i,j Indicate u i With v j The probability of forming a mapping between them.

[0067]

[0068] By u i v j Solving the relationship between the template image and the target image yields the optimal connection relationship between feature points. The higher the correlation weight between corresponding positions, the higher the probability of the associated feature points matching each other.

[0069] When measuring the accuracy of feature point matching results, it can be calculated through the positional correlation between pixels in the images. Assuming that most of the target object in the image or the object to be matched is a rigid target, the coordinate transformation between the associated point set of the target image and the template image can be completed using a fixed mapping matrix. Assuming the feature points PT in the target image... target Feature points PT within the template image template If a matching is achieved, then there exist matrices A and B such that the feature point PT... template The coordinates of the feature points PT are transformed through matrix operations. target :

[0070] PT target =PT template ×A+B

[0071] Where matrix A is a 2×2 matrix and matrix B is a 2×1 matrix:

[0072]

[0073]

[0074] Let feature point PT target The coordinates are x1, y1, and the feature point is PT. template If the coordinates of the two feature points are x2 and y2, then the mapping relationship between the two feature points is:

[0075]

[0076]

[0077]

[0078]

[0079] The calculation of the coordinate values ​​of each feature point in the target sample can then be expressed as:

[0080] x1 = a1x2 + b1y2 + c1

[0081] y1 = a²x² + b²y² + c²

[0082] Define matrices A and B as mapping matrices M for feature points, and then map the feature points PT... template With feature point PT target Solving for the matching results and verifying their accuracy will enable the following: Figure 3 The feature point matching results shown are as follows: Figure 3 The left image is the target image, and the right image is the template image. The line segments represent the lines connecting the matching feature points in the two images. Through the mapping matrix M, the target image can be... Figure 3 Points that are successfully matched in the right image are converted to coordinates in the left image and a one-to-one correspondence is achieved. At the same time, the coordinates of the corresponding fixed points in the template image can be mapped to the coordinates of the center point.

[0083] Feature point matching and feature vector extraction based on the Unet network structure can generate heatmap representations of feature points on high-resolution feature maps, improving the discriminability between feature points. However, prediction on high-resolution feature maps consumes significant computational resources, slowing down the overall image processing speed. This solution uses the target image and / or template image of the Siamese neural network as high-resolution images. After downsampling the input image to form a low-resolution image, the predicted heatmap is also a low-resolution map. To reduce computational resources, this solution uses a regression branch to introduce a biased prediction of feature point positions on the heatmap, thereby achieving high-resolution prediction results from a low-resolution map. Please refer to [link to relevant documentation]. Figure 2 .

[0084] Suppose a Siamese neural network predicts a heatmap at a 4x downsampling resolution. Because the predicted heatmap resolution is lower than the input image resolution, a highlighted area in the feature point heatmap branch will be represented as a pixel within a 4x4 region of the input image. To accurately match the results of the feature point heatmap branch to the template image, a bias is introduced into the predicted feature point coordinates (x, y) of the regression branch. The bias coefficient is 0 to 1. If it is a 4x downsampling, it is multiplied by a coefficient of 4, representing the specific offset position of the feature point within the downsampled region, thus achieving precise feature point localization.

[0085] Step 3: The Siamese neural network generates feature vector branches; based on the matched feature points, the feature vector branches are used to extract the feature vectors of the feature points.

[0086] In addition to the heatmap branch and the feature branch of the feature points, the twin neural network also includes a feature vector branch, which is used to extract feature vectors based on the feature points.

[0087] Assuming that in an input 512×512 resolution original image, feature points extracted from edge pixels generate a 128-dimensional feature vector, then convolution is needed to generate a 512×512×128 heatmap, which introduces a large amount of computation. This scheme downsamples the original image by a factor of 4 to generate a 128×128×128 heatmap, extracts features on the encoder and decoder, and then uses feature interpolation to extract the corresponding feature points from the heatmap, generating a feature vector.

[0088] In actual feature point detection, the number of detected feature points is usually in the range of 2000-3000, which is far less than 128×128. Therefore, using full convolution to predict feature vectors on the heatmap will generate a lot of computational redundancy. So, feature vector branching is introduced, and feature interpolation is used to limit the feature extraction operation to specified feature points, thereby achieving efficient feature vector extraction.

[0089] The feature vector branch uses a triplet loss function, randomly selecting k (k=8) samples from the feature points detected in the shared regions of the template and target images as positive samples, meaning k one-to-one corresponding feature points are selected from each pair of images; simultaneously, 128 feature points are randomly selected from the non-shared regions of the target and template images as negative samples. Cosine similarity is used to measure the feature similarity of each feature point, and the generation of the feature vector is supervised by setting the offset value of the triplet loss function to 0.6.

[0090] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for heterogeneous image matching based on twin neural networks, characterized in that: The method comprises the following steps: Step 1, obtaining a plurality of template images and corresponding target images; Step 2, inputting the template images and the corresponding target images into a twin neural network to generate a feature point heat map branch; determining the positions of the feature points using the feature point heat map branch, and matching the feature points in the target image to the feature points in the template image; In the step 2, the step of determining the positions of the feature points using the feature point heat map branch comprises: The encoder and the decoder in the twin neural network generate a heat map, the heat map is maximum-pooled at a fixed step, the pooled result is compared with the heat map, and points with the same values in the comparison are taken as candidate feature points; A confidence threshold is set, and the candidate feature points with values greater than the confidence threshold are taken as the feature points output by the feature point heat map branch, and the positions of the feature points are also output; In the step 2, the step of matching the feature points in the target image to the feature points in the template image comprises: The feature points output by the feature point heat map branch are taken as positive samples, feature points in a non-shared area of the template image and the target image are randomly selected as negative samples, and the feature similarities of the feature points are measured using cosine similarity; The global search method is used to calculate the correlation between the feature points in the target sample and the feature points in the template sample one by one to solve the global optimal matching relationship: where P i,j represents the element value mapped from the i-th row of the target sample to the j-th row of the template sample; H(P) is the entropy of P i,j ; Wherein, P represents a mapping of all rows of the target sample summing up to a vector a, all columns of the template sample summing up to a vector b, the mapping being U(a, b); C represents a cost of completing the element mapping of P; represents a regularization coefficient; There are P = u i K i,j v j , u i and v j represent a set of solutions obtained by mapping, u i is the displacement of the i-th row of feature points on the target image, and v j is the displacement of the j-th row of feature points on the template image; By solving the relationship between u i , v j , the optimal connection relationship between the feature points in the template image and the feature points in the target image is obtained. After the step of matching the feature points in the target image to the feature points in the template image in the step 2, the step of verifying the accuracy of the matching result is further included: Setting a feature point PT within a target image target Matching with a feature point PT within a template image template Matching, there exist a matrix A, a matrix B such that the coordinate value of the feature point PT template is changed into the coordinate value of the feature point PT target by matrix operation: Wherein, matrix A is a 2x2 matrix, and matrix B is a 2x1 matrix: Let the coordinate value of the feature point PT target be x1, y1, and the coordinate value of the feature point PT template be x2, y2, then the mapping relationship between the two feature points is: Definition of matrix A, matrix B is the mapping matrix M of feature points, through the feature points PT template With the feature points PT target The solution of the matching result accuracy; Step 3, the twin neural network generates a feature vector branch; according to the matched feature points, the feature vector branch is used to extract the feature vectors of the feature points.

2. The twin neural network based multi-modal image matching method of claim 1, wherein: The step 1 specifically comprises the following steps: taking the whole image of an original image as a target image, taking a local cut image of the original image as a template image, and the resolution of the target image and the template image is equal or unequal; the template image and the target image are data-augmented in the color, angle and contrast manner; and the shared area of the augmented template image and the target image is taken as the input of the twin neural network.

3. The twin neural network based multi-modal image matching method of claim 1, wherein: In the step 2, the step of using the regression branch to predict the bias of the position coordinates (x, y) of the feature points is further included, and the bias coefficient is 0-1, so as to improve the accuracy of the feature point position.

4. The twin neural network based multi-modal image matching method of claim 1, wherein: In the step 3, according to the matched feature points, the step of using the feature vector branch to extract the feature vectors of the feature points comprises: extracting features from the encoder and the decoder, and performing feature difference on the feature points, so as to generate the feature vectors.

Images