Image decoupling and matching method based on invariant features and variable features

Abstract

The invention discloses an image decoupling and matching method based on invariant features and variable features. The image decoupling method comprises the following steps: constructing an image decoupling network based on the invariant features and the variable features; selecting any two to-be-matched image blocks in the training data set, and respectively inputting the two to-be-matched image blocks into the network for decoupling to obtain variable feature data and invariable feature data; enabling the distance between the invariant feature data output by the invariant feature extraction network to reach a first preset minimum value through a pre-constructed first constraint condition; fusing and reconstructing the variable feature data and the invariable feature data of the same image block to be matched to obtain a reconstructed image; enabling the distance between the reconstructed image and the to-be-matched image block to reach a second preset minimum value through a pre-constructed second constraint condition; and through iterative training, a trained image decoupling network is obtained so as to be used for decoupling the invariant features and the variable features of the image. According to the method, the decoupled invariant features contain more matchable information, so that the matching effect can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image decoupling and matching method based on invariant features and variable features. Background Art

[0002] Image feature matching aims to establish accurate pixel correspondence between images, and is an important prerequisite for many subsequent computer vision applications, such as image retrieval, image stitching, image positioning, and 3D reconstruction. The typical image feature matching process is: image key point detection, image feature description, and image feature matching. With the development of deep learning, more and more end-to-end image feature matching methods have been developed in recent years. The end-to-end image feature matching process is to pass in a matchable image and directly obtain the image matching result through network prediction.

[0003] Traditional image feature description algorithms are usually constructed subjectively by designers. These algorithms are usually mathematical descriptions of local information around key points in the image. For a long time, manually designed descriptors have been the mainstream research method in this field until the emergence of deep learning technology. Since the advent of deep learning technology, various research fields that have applied this technology have made significant progress, and the field of computer vision is no exception.

[0004] Compared with traditional manual methods, deep learning methods are data-driven and can better adapt to challenging environments and handle more complex and even extreme scenarios. By carefully designing the network structure, reasonably setting the loss function, and making full use of the data set for training, the image feature description method based on deep learning can more easily obtain feature descriptors that are conducive to establishing matching. With the continuous advancement of deep learning, the end-to-end image feature matching network has been able to bypass the traditional feature descriptor learning process and directly obtain matching results from the image block data to be matched, so that it is no longer restricted by the quality of key points. The results obtained by this method are more accurate.

[0005] Although most of these deep learning-based feature description methods perform well, they are usually affected by various interferences in the image, such as illumination changes, perspective changes, etc. After feature extraction is disturbed by these noisy information, the extracted features not only contain invariant information that can be matched, but also carry interfering variable information, resulting in impure features, which affects the matching effect. Summary of the invention

[0006] The main purpose of the present invention is to provide an image decoupling and matching method based on invariant features and variable features, which can directly extract invariable feature data in images under different interference environments, thereby improving the image matching accuracy.

[0007] The technical solution adopted by the present invention is:

[0008] A method for decoupling an image based on invariant features and variable features is provided, comprising the following steps:

[0009] Constructing an image decoupling network based on invariant features and variable features, which includes a variable feature extraction network and an invariant feature extraction network;

[0010] Select any two image blocks to be matched in the training data set, input them into the variable feature extraction network and the invariant feature extraction network respectively, and decouple them to obtain variable feature data and invariant feature data;

[0011] The distance between the invariant feature data output by the invariant feature extraction network reaches a first preset minimum value through a pre-constructed first constraint condition;

[0012] The variable feature data and the invariant feature data obtained by decoupling the same image block to be matched are fused and reconstructed to obtain a reconstructed image block; a second constraint condition constructed in advance is used to make the distance between the reconstructed image block and the image block to be matched reach a second preset minimum value;

[0013] Through iterative training, a trained image decoupling network based on invariant features and variable features is obtained to be used for decoupling the invariant features and variable features of the image.

[0014] Following the above technical solution, the first constraint is the triplet loss function and its variants determined by algorithms such as HardNet and SOSNet.

[0015] Following the above technical solution, the second constraint condition is a reconstruction constraint loss function constructed according to the relationship between the reconstructed image block and the image block to be matched.

[0016] Following the above technical solution, depending on the training data set, the image block to be matched can be a local image block cropped around the key point or a complete image of the original size.

[0017] The triplet involved in the first constraint condition is constructed as follows: the triplet contains 3 image blocks, 2 of which should come from different images and have a matching relationship between them; the other image block has no matching relationship with the above two image blocks.

[0018] Following the above technical solution, a training sample set is constructed based on the Brown data set. First, determine which two images contain a common scene and construct a set of image pairs to be matched. Secondly, detect key points on each pair of images to be matched, and mark the matching relationship of the key points. Based on this, iteratively optimize the image decoupling network parameters to minimize the first constraint and the second constraint, that is, the invariant features of the matching key points are as similar as possible and the image reconstructed by the combination of the invariant features and the variable features is as similar as possible to the input image. For the full size, then traverse each pair of matching key points to construct a triple. Specifically, select a pair of matching key points, crop them with a fixed size as the center to obtain two image blocks, and then randomly select one from the other key points, and crop the third image block according to the same steps as above. The multiple triplets constructed are used as the training data set for the image decoupling network; each training takes minimizing the average loss of the triples as the training goal.

[0019] Following the above technical solution, the method further includes the steps of: testing the image decoupling network using a test set, and performing a multi-index evaluation on the descriptors of the variable feature data and the invariant feature data output by the image decoupling network using a preset evaluation method.

[0020] Following the above technical solution, the test set is specifically divided into difficulty levels according to the difference in shooting angles between the image pairs in the data set or divided into difficulty levels according to the baseline width between the image pairs and then tested to reflect the performance of the method of the present invention on data sets of different difficulty levels.

[0021] Following the above technical solution, the reconstructed image block is specifically reconstructed through an image reconstruction network, and the image reconstruction network includes multiple deconvolution modules.

[0022] The present invention also provides an image decoupling system based on invariant features and variable features, comprising:

[0023] An image decoupling network construction module is used to construct an image decoupling network based on invariant features and variable features, which includes a variable feature extraction network and an invariant feature extraction network;

[0024] A training module is used to select a triplet of image blocks to be matched (two matching image blocks and one non-matching image block) in a training data set, and input them into a variable feature extraction network and an invariant feature extraction network to obtain variable feature data and invariant feature data of each image block; and make the triplet loss function reach a first preset minimum value through a pre-constructed first constraint condition;

[0025] A reconstruction module, used for fusing and reconstructing the variable feature data and the invariant feature data obtained from the same input image block to obtain a reconstructed image block; and making the distance between the reconstructed image block and the input image block reach a second preset minimum value through a pre-constructed second constraint condition;

[0026] The iterative module is used to obtain a trained image decoupling network based on invariant features and variable features through iterative training, so as to be used for decoupling the invariant features and variable features of the image.

[0027] The present invention also provides an image matching method based on decoupling of invariant features and variable features, comprising the following steps:

[0028] For any pair of images to be matched, extract all key points in the image, extract image blocks of a certain size with the key points as the center and perform normalization processing;

[0029] The normalized image blocks are respectively input into a pre-trained image decoupling network based on invariant features and variable features to extract the invariant feature data of each image block, that is, the invariant features of the corresponding key points; the image decoupling network based on invariant features and variable features is obtained according to the image decoupling method based on invariant features and variable features according to claim 1;

[0030] For a key point on the first image, according to the invariant feature data extracted above, find the key point with the closest distance on the other image as a pair of matching key points;

[0031] Traverse all key points on the first image and repeat the above steps to complete the matching of the image pairs to be matched.

[0032] The present invention also provides an image matching system based on decoupling of invariant features and variable features, comprising:

[0033] A key point extraction module is used to extract all key points in a pair of images to be matched, extract image blocks of a certain size with the key points as the center, and perform normalization processing;

[0034] Image decoupling, for inputting the normalized image blocks into a pre-trained image decoupling network based on invariant features and variable features, respectively, to extract the invariant feature data of each image block as the invariant feature of the corresponding key point; the image decoupling network based on invariant features and variable features is obtained according to the image decoupling method based on invariant features and variable features according to claim 1;

[0035] The matching module is used to complete the matching of all key points on the image pair to be matched according to the invariant features of each key point.

[0036] The present invention also provides a computer storage medium, which stores a computer program that can be executed by a processor, and the computer program executes the decoupled image matching method based on invariant features and variable features described in the above technical solution.

[0037] The beneficial effects of the present invention are as follows: in order to solve the problem that there are differences in perspective, scale, illumination, rotation, etc. between different views in a multi-view matching task, the present invention proposes a dual-branch feature decoupling network, in which one branch focuses on learning the variable features introduced by external environmental factors in each image, and the other branch is used to learn the invariant characteristics between image pairs that are independent of external factors. The decoupled invariant features separate the external difference factors and contain more matchable information, thereby improving the matching effect; in order to further constrain the feature decoupling process, the present invention proposes an image reconstruction loss, that is, the variable feature data and the invariant feature data obtained from the same input image block are fused and reconstructed to obtain a reconstructed image block, and the distance between the reconstructed image block and the input image block is made to reach a second preset minimum value through pre-constructed constraints, thereby further calculating the reconstruction constraints to ensure the validity of the learned variable features and invariant features.

[0038] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 is a flow chart of an image decoupling method based on invariant features and variable features according to an embodiment of the present invention;

[0041] Figure 2 It is a flowchart of an image decoupling method based on invariant features and variable features according to an embodiment of the present invention;

[0042] Figure 3 A flowchart of an image decoupling method based on invariant features and variable features according to an embodiment of the present invention (based on an image block descriptor learning network);

[0043] Figure 4 A flowchart of an image decoupling method based on invariant features and variable features according to an embodiment of the present invention (based on a dense descriptor learning network);

[0044] Figure 5 A schematic diagram of a flow chart of an image decoupling method based on invariant features and variable features according to an embodiment of the present invention (based on an end-to-end image matching network);

[0045] Figure 6 This is a schematic diagram of the structure of an image decoupling system based on invariant features and variable features according to an embodiment of the present invention;

[0046] Figure 7 This is a flow chart of a decoupled image matching method based on invariant features and variable features according to an embodiment of the present invention;

[0047] Figure 8 The figure is a schematic diagram of the structure of a decoupled image matching system based on invariant features and variable features according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] It should be noted that the illustrations provided in the embodiments of the present invention are only used to illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout type may also be more complicated.

[0050] In the present invention, it is also necessary to explain that, if the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, if the terms "first" and "second" appear, they are only used for description and distinction purposes, and cannot be understood as indicating or implying relative importance.

[0051] The image decoupling method based on invariant features and variable features of the present invention mainly utilizes the separation of variables and invariants to better capture the key features required for establishing matching in common information, making the network more reliable and versatile, and able to adapt to the differences between various image pairs, thereby improving the effect of the matching task; the present invention has universal applicability and can be easily integrated into many existing frameworks, and the matching results obtained in actual data can achieve a significant improvement in accuracy, thereby achieving an improvement in comprehensive performance.

[0052] like Figure 1 As shown, the image decoupling method based on invariant features and variable features in an embodiment of the present invention includes the following steps:

[0053] S101, constructing an image decoupling network based on invariant features and variable features, which includes a variable feature extraction network and an invariant feature extraction network;

[0054] S102, selecting a triplet (two matching image blocks and one non-matching image block) of image blocks to be matched in the training data set, and inputting them into a variable feature extraction network and an invariant feature extraction network respectively to obtain variable feature data and invariant feature data of each image block;

[0055] S103, using a pre-constructed first constraint condition to ensure that the invariant feature data output by the invariant feature extraction network meets a first preset minimum value;

[0056] S104, fusing and reconstructing the variable feature data and the invariant feature data of the same image block to be matched to obtain a reconstructed image; and making the distance between the reconstructed image block and the image block to be matched reach a second preset minimum value through a pre-constructed second constraint condition;

[0057] S105. Through iterative training, a trained image decoupling network based on invariant features and variable features is obtained for decoupling the invariant features and variable features of the image.

[0058] According to different training data sets, the image block to be matched is an image block cropped at a certain size with the key point as the center, or a complete shot image of the original size.

[0059] like Figure 2 As shown, the learning network for image decoupling based on invariant features and variable features includes: an invariant feature extraction network, a variable feature extraction network, a variable and invariant fusion module and an image reconstruction network.

[0060] The invariant feature extraction network and the variable feature extraction network are used to extract features from images. The former focuses on extracting invariant features that can match the image itself, and the latter focuses on extracting variable features of the environment in which the image is located. The variable and invariant fusion module is used to fuse the variable and invariant features and output features for reconstructing the image. The image reconstruction network is used to reconstruct and learn the fused features to obtain a reconstructed image. Based on the loss function, a reconstruction loss constraint on the reconstructed image is added after the constraint item of the invariant feature extraction network.

[0061] The following is explained through specific examples:

[0062] Example 1

[0063] Embodiment 1 provided by the present invention is an image decoupling method for variable features and invariant features, which is based on an image block descriptor learning network. The image block to be matched is cut into an image block of a specified size with the key point as the center, and the network learns a descriptor for each image block. Networks based on image block learning descriptors such as HardNet, SOSNet, L2Net, GeoDesc, etc. can be used as backbone networks. This embodiment is specifically described using the HardNet (Working hard to know your neighbor's margins: Local descriptor learning loss) algorithm as an example. The HardNet algorithm is based on the network structure of L2Net to learn the descriptor of the image block. HardNet proposes the famous triplet loss (Tripletloss), which requires that the feature distance of the matching image block and the feature distance of the non-matching image block be separated by a certain margin. At the same time, the network pays more attention to those negative samples (non-matching image blocks) that are difficult to distinguish, thereby making the features more distinguishable.

[0064] like Figure 3 As shown, this embodiment includes the following steps:

[0065] Step 1: construct a training sample set and a test sample set based on the captured image data.

[0066] Step 1 is the data preparation stage, which prepares the data set for network training and testing. First, a matching pair of images is constructed based on whether the two images contain the same scene; second, key points are detected for each image based on a certain key point detection method, and these key points are matched to obtain matching label information for subsequent use.

[0067] In a possible embodiment, step 1 includes:

[0068] Step 101, detect key points on any pair of matching images and build a matching relationship between them.

[0069] In specific implementation, different key point detection algorithms can be used, such as SIFT algorithm, Surf algorithm, SuperPoint algorithm, R2D2, etc. In addition to obtaining the key point position, this type of method can also obtain the feature descriptor of the key point position at the same time. The initial matching relationship between key points is established based on the similarity of the feature descriptors, and then combined with manual screening to obtain accurate matching label information.

[0070] Step 102: With each key point as the center, cut an image block of size w×w, where w can be 32 or 64, etc. The specific value depends on the selected method. According to the matching label information in step 101, select 2 matching image blocks and 1 non-matching image block to construct a triple, and multiple triplets constitute a training sample set.

[0071] Step 103: divide the data set into test sets of multiple categories according to the matching difficulty between image pairs.

[0072] In the specific implementation, the training data set selected in this embodiment includes but is not limited to the Brown data set, which has 3 scenes. In each scene, several image pairs are selected to construct triples according to the above method as the training data set. The test set selected in this embodiment includes but is not limited to HPatches, COLMAP, MegaDepth, ScanNet and other data sets. Since the degree of difference between the images of each scene (such as viewing angle, etc.) is large or small, the data set can be divided into multiple categories of data sets according to the matching difficulty between the image pairs to test the effectiveness of this method in different difficulty scenes. Specifically, for HPatches, the first image in each scene and the subsequent five images are directly combined into 5 pairs of image pairs with gradually increasing viewpoint differences. There are a total of 116 scenes, so there are a total of 580 pairs of image pairs. For COLMAP and MegaDepth, the entire data set can be divided into three categories: easy (0°-15°), moderate (15°-30°), and hard (30°-60°) according to the difference in the sight angle between the image pairs. For ScanNet, the entire dataset can be divided into three categories: easy (10 frames), moderate (30 frames), and hard (60 frames) according to the frame interval between image pairs.

[0073] Step 2, construct a descriptor learning network based on variable and invariant decoupling and an image feature matching loss function with a reconstruction loss constraint; the descriptor learning network based on variable and invariant decoupling includes: an invariant feature extraction network, a variable feature extraction network, a variable and invariant fusion module and an image reconstruction network; in this example, HardNet is selected as the backbone network.

[0074] The invariant feature extraction network and the variable feature extraction network are used to extract features from images. The former focuses on extracting the invariant features of two matching images, while the latter focuses on extracting the variable features of images caused by factors such as camera viewpoint and environment. The variable and invariant fusion module is used to fuse the variable and invariant features and output features for reconstructing images. The image reconstruction network is used to reconstruct and learn the fused features.

[0075] The input of the invariant feature extraction network and the variable feature extraction network are both image blocks, and the outputs are the invariant features and variable features of the image blocks respectively.

[0076] like Figure 2 As shown, in order to reduce the number of network parameters, the invariant feature extraction networks of the two images share weights, and the variable feature extraction networks also share weights. The input of both networks is an image block, and the output is a feature map, whose length and width are the same as the input image block. The backbone network module of the present invention has many optional schemes, and a suitable backbone network can be selected according to needs.

[0077] In a possible implementation manner, the variable feature extraction network and the invariant feature extraction network both include 7 convolution modules.

[0078] The first six convolutional modules include: a 3×3 convolution, a batch normalization layer, and an activation function. The padding in the 1st, 2nd, 4th, and 6th convolutional modules is 1, and the padding in the 3rd and 5th convolutional modules is The seventh convolution module includes an 8×8 convolution, a batch normalization layer, and an activation function.

[0079] In a possible implementation manner, the variable and invariant fusion module includes an element-wise multiplication, a 3×3 convolution, a batch normalization layer, and an activation function, followed by an upsampling and a maximum pooling, and finally element-wise addition with the original features to obtain the final fused features.

[0080] The image reconstruction network reconstructs and learns the input features to obtain the reconstructed image.

[0081] In one possible implementation manner, the image reconstruction network consists of four deconvolution modules, wherein the first three deconvolution modules each contain a 3×3 convolution, a batch normalization, and a ReLU layer, with a stride of 2 and a padding of 1, and the convolution kernel size in the last deconvolution module is 1×1, with a stride of 1 and a padding of 0.

[0082] The image feature matching loss function with reconstruction constraint term constrains the network to accurately restore the image block to be matched, thereby ensuring that the decoupled learning process does not lose effective information, and the separated variable feature information and invariant feature information have practical restoration significance. The reconstruction loss constraint term proposed in the present invention can be combined with the loss function of the current mainstream method.

[0083] In a possible implementation manner, a reconstruction constraint loss term L for the image reconstruction network is constructed by combining the basic network module selected in this embodiment and the designed learning network for reconstructing the image. recon for:

[0084]

[0085] Where i represents the i-th pixel in the image.

[0086] In particular, the reconstruction loss form here is not only the formula (5), but also the following reconstruction loss form:

[0087]

[0088] Formula (5) uses the mean square error between pixels as the loss term, and formula (6) adds SSIM to the first one. The latter is stricter. If you want stronger constraints, you can choose the latter, otherwise you can choose the former.

[0089] In the specific implementation, this embodiment uses the triple loss L triplet To constrain the similarity of image block features, the specific form of the triplet loss function can be found in HardNet. The overall loss function is: L total =L triplet +λ recon L recon , where in order to balance the triplet loss and reconstruction loss and achieve the best experimental results, λ is set according to experience in practice recon =0.01.

[0090] Step 3: As described in step 102, multiple triplets are constructed based on the Brown dataset as network training datasets. The decoupled variable and invariant learning network is trained based on the selected training dataset. The input of each training is N triplets, where N is a preset batch-size, and the output is the features of all image blocks, and the aforementioned L is minimized. total As a training objective, the trained image descriptor learning network based on decoupling of variables and invariants is obtained.

[0091] In the network training phase, the network of the present invention is trained based on the Brown data set. In training, an epoch is a complete training of the model using all the data of the selected training data set, a batch is a small sample of the training data set that performs a back-propagation parameter update on the model weights, and an iteration is the process of updating the parameters of the model using a batch of data. The present invention can select different training results, such as using the model of the last iteration as the final model, or adding a validation set during training to select the model with the best validation result as the final model, or using multiple epochs for training and then selecting the best model as the result. At the same time, batch-size also has a certain impact on the network training results, so the network hyperparameters can be adjusted according to the network training results.

[0092] In the embodiment of the present invention, a multi-epoch training method can be used, the number of epochs is 20, the batch-size is set to 6, the optimizer is ADAM, and the learning rate is 10 -4 .

[0093] In the specific implementation, real scenes such as 3D reconstruction, pose estimation, image matching, etc. Use the current popular evaluation methods to evaluate multiple indicators, such as homography matrix estimation, relative pose estimation accuracy, etc.

[0094] In a possible embodiment, the evaluation method is used to evaluate the descriptor performance using multiple indicators, including: using homography matrix estimation accuracy to evaluate the performance in downstream tasks such as planar scenes and image stitching. Referring to the SuperPoint method, the correctness of the homography matrix evaluation between each pair of images is defined by the following formula:

[0095]

[0096] Among them, e represents the given pixel threshold, and x i They are the positions of the four corner points of the source image transformed by the algorithm estimation and the true homography matrix respectively.

[0097] In a possible embodiment, the evaluation method is used to evaluate the descriptor performance using multiple indicators, including: using relative pose estimation accuracy (Rotation Estimation Accuracy and Translation Estimation Accuracy) to evaluate the performance in the downstream task of 3D reconstruction of non-planar scenes, and calculating the angle error based on the rotation matrix and the actual rotation matrix. The calculation rule is:

[0098]

[0099] in, Represents the angle prediction value, R gt Indicates the true value of the angle.

[0100] When the angle θ is less than a given threshold, the estimation is considered to be wrong. The translation is also estimated using the vector angle error, because existing algorithms can only calculate the relative scale of the translation vector rather than the absolute scale.

[0101] Example 2

[0102] Embodiment 2 provided by the present invention is another embodiment of an image decoupling method based on variable and invariant features provided by the present invention. Embodiment 2 is based on a dense descriptor learning network.

[0103] This embodiment uses a dense pixel-based descriptor learning network based on variable and invariant feature decoupling, which learns a descriptor for each pixel in the entire image block to be matched. The present invention uses the CAPS (Learning Feature Descriptors using Camera Pose Supervision) algorithm as an example of the backbone network for specific explanation. CAPS proposes a weak supervision framework that only contains feature descriptions from coarse to fine. The most important contribution of this work is that it proposes to use relative camera poses directly as weak supervision labels, thereby avoiding the labeling of a large number of training data sets.

[0104] like Figure 4 As shown, the method comprises the following steps:

[0105] Step 1: construct a matchable image pair based on whether the two images contain the same scene, where the image pair includes two images; perform key point detection on the two images respectively to obtain a first set of feature points and a second set of feature points, and construct a data set based on the image pair and the first set of feature points and the second set of feature points.

[0106] Step 1 is the data preparation stage, which prepares the data set for network training and testing. First, the image size in the data set is normalized; second, the key points are detected for each image based on a certain key point detection method, and the matching relationship between the key points is established; finally, based on specific classification rules, all the data is classified and organized for subsequent use.

[0107] In a possible embodiment, step 1 includes:

[0108] Step 101', normalizing the size of each image to obtain an image of a set size, and adjusting the intrinsic and extrinsic parameters of each image according to the scaling ratio to obtain the true value of the scaled image again;

[0109] In the specific implementation, the data set used by the CAPS algorithm selected in this embodiment is composed of pairs of images containing the same scene. For the convenience of training, the image size can be normalized to 640×480.

[0110] Step 102 ′: randomly select a number of pairs of matching images to construct a training data set.

[0111] In a specific implementation, the first image I in the image pair is obtained. 1 The m feature points of the image pair form the first set of feature points, and the second image I2 The n feature points of the image are combined into a second set of feature points, and the matching relationship between them is established. Different key point detection algorithms can be used, such as SIFT algorithm, Surf algorithm, SuperPoint algorithm, R2D2, etc. In addition to obtaining the key point position, this type of method can also obtain the feature descriptor of the key point position at the same time. The initial matching relationship between the key points is established based on the similarity of the feature descriptors, and then combined with manual screening to obtain accurate matching label information.

[0112] Step 103 ′, dividing the data set into multiple categories according to the difficulty of matching between image pairs.

[0113] The data sets selected in this embodiment include, but are not limited to, HPatches, COLMAP, MegaDepth, ScanNet and other data sets. Since the degree of difference (such as viewing angle, etc.) between the images of each scene varies, the data set can be divided into multiple categories of data sets according to the difficulty of matching between the image pairs, so as to test the effectiveness of the method under different difficulty scenes. Specifically, for HPatches, the first image in each scene and the subsequent five images are directly combined into 5 pairs of image pairs with gradually increasing viewpoint differences. There are a total of 116 scenes, so there are a total of 580 pairs of image pairs. For COLMAP and MegaDepth, the entire data set can be divided into three categories: easy (0°-15°), moderate (15°-30°), and hard (30°-60°) according to the difference in sight angles between the image pairs. For ScanNet, the entire data set can be divided into three categories: easy (10 frames), moderate (30 frames), and hard (60 frames) according to the frame number interval between the image pairs.

[0114] Step 2, construct a descriptor learning network based on variable and invariant decoupling and an image feature matching loss function with a reconstruction loss constraint; the descriptor learning network based on variable and invariant decoupling includes: an invariant feature extraction network, a variable feature extraction network, a variable and invariant fusion module and an image reconstruction network; in this example, the CAPS algorithm is selected as the backbone network.

[0115] The invariant feature extraction network and the variable feature extraction network are used to extract features from images. The former focuses on extracting invariant features that can match images, while the latter focuses on extracting variable features caused by factors such as camera viewpoints and environments. The variable and invariant fusion module is used to fuse the variable and invariant features and output features for reconstructing images. The image reconstruction network is used to reconstruct and learn the fused features.

[0116] The input of the invariant feature extraction network and the variable feature extraction network are both images, and the outputs are the invariant features and variable features of the images respectively.

[0117] In order to reduce the number of network parameters, the invariant feature extraction network of the two images shares weights, and the variable feature extraction network also shares weights. The input of both networks is a complete image, and the output is a feature map of each pixel, whose length and width are the same as the input image size. The backbone network module of the present invention has many optional schemes, and a suitable backbone network can be selected according to demand.

[0118] In a possible implementation manner, the variable feature extraction network and the invariant feature extraction network both include 5 convolution modules.

[0119] The first convolution module includes a 7×7 convolution, a batch normalization and an activation function. The second to fourth convolution modules are composed of the first three layers of ResNet50. The fifth convolution module includes a 1×1 convolution, a batch normalization and an activation function.

[0120] In a possible implementation manner, the variable and invariant fusion module includes an element-wise multiplication, a 3×3 convolution, a batch normalization layer, and an activation function, followed by an upsampling and a maximum pooling, and finally element-wise addition with the original features to obtain the final fused features.

[0121] The image reconstruction network reconstructs and learns the input features to obtain the reconstructed image.

[0122] In one possible implementation manner, the image reconstruction network consists of four deconvolution modules, wherein the first three deconvolution modules each contain a 3×3 convolution, a batch normalization, and a ReLU layer, with a stride of 2 and a padding of 1, and the convolution kernel size in the last deconvolution module is 1×1, with a stride of 1 and a padding of 0.

[0123] The image feature matching loss function with reconstruction constraint term constrains the network to accurately restore the image block to be matched, thereby ensuring that the decoupled learning process does not lose effective information, and the separated variable feature information and invariant feature information have practical restoration significance. The reconstruction loss constraint term proposed in the present invention can be combined with the loss function of the current mainstream method.

[0124] In a possible implementation manner, a reconstruction constraint loss term L for the image reconstruction network is constructed by combining the basic network module selected in this embodiment and the designed learning network for reconstructing the image. recon for:

[0125]

[0126] Where i represents the i-th pixel in the image.

[0127] In particular, the reconstruction loss form here is not only the formula (9), but also the following reconstruction loss form:

[0128]

[0129] In a possible embodiment, the reconstruction constraint loss term L recon With the weight λ set recon The image matching loss function based on variable and invariant decoupling is formed by combining with the loss function of the invariant feature extraction network itself.

[0130] When the camera pose is known, epipolar geometry is a very useful information. This loss requires that the correct correspondence of the query keypoint in one image must lie on the corresponding epipolar line in the other image. In other words, if the distance from the matching point to the epipolar line is greater than a threshold, it is more likely to be an incorrect match. Epipolar loss L ep It is required that the correct correspondence φ(p) of the query key point in the image must be located on the corresponding epipolar line Fp in the other image, which has the following form:

[0131] L ep (p) = dist(φ(p), Fp) # (11)

[0132] Epipolar geometry is a loose constraint because it only constrains the correct match point to be on the epipolar line, but cannot determine its specific location. In this case, the network cannot distinguish between incorrect matches that slide along the epipolar line. Therefore, another cycle consistency loss L is applied to the correct match in the network. cycle , the specific form is as follows:

[0133]

[0134] In the specific implementation, this embodiment uses the cycle consistency loss L cycle And the polar loss L ep To constrain the similarity of image features, the cycle consistency loss L cycle That is, for the query key point p, its predicted matching point p on the image to be matched is required m And the matching point p′ found on the image block to be matched m As close as possible, and the overall loss function is: L total =L cycle +L ep +λ recon L recon In order to balance the effects of reconstruction loss and other losses, λ is set according to experimental results and experience in practice. recon=0.01.

[0135] Step 3, based on the size of the overlap value of the image pairs in the Megadepth dataset, they are divided into three categories: simple, general and difficult, and then a certain number of image pairs are randomly selected from the three categories in a ratio of 1:1:1 to form a training dataset, and all images are further uniformly scaled to a grayscale image of 640×640 size, and the decoupled variables and invariants learning network is trained based on the training dataset. The input of the network is the entire training dataset, and the output is a dense feature representation of the corresponding grayscale image, so as to obtain the trained image descriptor learning network based on variable and invariant decoupling.

[0136] In the network training phase, a training data set is constructed based on the MegaDepth data set to train the network of the present invention. In training, an epoch is a complete training of the model using all the data in the training data set, and a batch is a small sample of the training data set that performs a back-propagation parameter update on the model weights, and iteration is the process of updating the parameters of the model using a batch of data. The present invention can select different training results, such as using the model of the last iteration as the final model, or adding a validation set during training to select the model with the best validation result as the final model, or using multiple epochs for training and then selecting the best model as the result. At the same time, batch-size also has a certain impact on the network training results, so the network hyperparameters can be adjusted according to the network training results.

[0137] In the embodiment of the present invention, a multi-epoch training method can be used, the number of epochs is 10 times, the batch-size is set to 1024, the optimizer is SGD, and the learning rate is 10.0.

[0138] In a possible embodiment, step 3 further includes:

[0139] Step 4, construct a test set based on the data set, that is, including but not limited to HPatches, COLMAP, MegaDepth, ScanNet and other data sets to construct a test set. Specifically, for HPatches, the first image in each scene and the subsequent five images are directly combined into 5 pairs of image pairs, with a total of 116 scenes, so there are a total of 580 pairs of image pairs. For COLMAP and MegaDepth, the entire data set can be divided into three categories according to the angle difference between the image pairs: easy (0°-15°), moderate (15°-30°), and hard (30°-60°). For ScanNet, the entire data set can be divided into three categories according to the frame number interval between the image pairs: easy (10 frames), moderate (30 frames), and hard (60 frames). Based on the test set, the trained descriptor learning network based on the decoupling of variables and invariants is applied to the real scene for testing to obtain the descriptor performance; the descriptor performance is evaluated by multiple indicators using the evaluation method.

[0140] In the specific implementation, real scenes such as 3D reconstruction, pose estimation, image matching, etc. Use the current popular evaluation methods to evaluate multiple indicators, such as homography matrix estimation, relative pose estimation accuracy, etc.

[0141] In a possible embodiment, the evaluation method is used to evaluate the descriptor performance using multiple indicators, including: using homography matrix estimation accuracy to evaluate the performance in downstream tasks of planar scenes. Referring to SuperPoint, the correctness of the homography matrix evaluation between each pair of images is defined by the following formula:

[0142]

[0143] Among them, e represents the given pixel threshold, and x i They are the four corner points of the source image transformed by the algorithm estimation and the true homography matrix respectively.

[0144] In a possible embodiment, the evaluation method is used to evaluate the descriptor performance using multiple indicators, including: using relative posture estimation accuracy (Rotation Estimation Accuracy and Translation Estimation Accuracy) to evaluate the performance in downstream tasks of non-planar scenes, and calculating the angle error based on the rotation matrix and the actual rotation matrix. The calculation rule is:

[0145]

[0146] in, Represents the angle prediction value, R gt Indicates the true value of the angle.

[0147] When the angle θ is less than a given threshold, the estimation is considered to be wrong. The translation is also estimated with angle error, because the existing algorithm can only calculate the relative scale of the translation vector instead of the absolute scale.

[0148] Example 3

[0149] Embodiment 3 provided by the present invention is another embodiment of an image decoupling method based on variable and invariant features provided by the present invention, which is mainly based on an end-to-end image learning network.

[0150] This embodiment uses an end-to-end image matching learning network based on the decoupling of variable and invariant features, which directly generates matching results for image pairs. The present invention uses the LoFTR (Detector-Free Local Feature Matching with Transformers) algorithm as an example to specifically illustrate. LoFTR introduces the Transformer module into the feature learning network for the first time. It does not require a key point detector, and its performance is significantly ahead of other methods.

[0151] like Figure 5 As shown, the method specifically includes:

[0152] Step 1 is the data preparation stage, which prepares the data set for network training and testing. First, the image size in the data set is normalized, and the true value information of each image is processed synchronously; secondly, based on specific classification rules, all data are classified and organized for subsequent use.

[0153] In a possible embodiment, step 1 includes:

[0154] Step 101', normalizing the size of each image to obtain an image of a set size, and adjusting the intrinsic and extrinsic parameters of each image according to the scaling ratio to obtain the true value of the scaled image again;

[0155] In a specific implementation, the data set used by the LoFTR algorithm selected in this embodiment is composed of pairs of randomly selected image pairs containing the same scene. For the convenience of training, the image size can be normalized to 640×640.

[0156] Step 102 ′, classify the scenes into categories, randomly select two images of the same category as an image pair, and select a set number of image pairs in each category to generate a training data set.

[0157] Step 103 ′, dividing the data set into test sets of multiple categories according to the difficulty of matching between image pairs.

[0158] In the specific implementation, the data sets selected in this embodiment include but are not limited to HPatches, COLMAP, MegaDepth, ScanNet and other data sets. Due to factors such as the difference in viewing angles between the images of each scene, the data set can be divided into multiple categories of data sets according to the difficulty of matching between image pairs. Specifically, for HPatches, the first image in each scene and the subsequent five images are directly combined into 5 pairs of image pairs, with a total of 116 scenes, so there are a total of 580 pairs of image pairs. For COLMAP and MegaDepth, the entire data set can be divided into three categories: easy (0°-15°), moderate (15°-30°), and hard (30°-60°) according to the angle difference between the image pairs. For ScanNet, the entire data set can be divided into three categories: easy (10 frames), moderate (30 frames), and hard (60 frames) according to the frame number difference between the image pairs.

[0159] Step 2, construct an image matching learning network based on variable and invariant decoupling and an image feature matching loss function with a reconstruction loss constraint; the image matching learning network based on variable and invariant decoupling includes: an invariant feature extraction network, a variable feature extraction network, a variable and invariant fusion module and an image reconstruction network; in this example, the LoFTR algorithm is selected as the backbone network.

[0160] The invariant feature extraction network and the variable feature extraction network are used to extract features from images. The former focuses on extracting invariant features that can match images, while the latter focuses on extracting variable features caused by factors such as camera viewpoints and environments. The variable and invariant fusion module is used to fuse the variable and invariant features and output features for reconstructing images. The image reconstruction network is used to reconstruct and learn the fused features.

[0161] The input of the invariant feature extraction network and the variable feature extraction network are both images, and the outputs are the invariant features and variable features of the images respectively.

[0162] In order to reduce the number of network parameters, the invariant feature extraction network of the two images shares weights, and the variable feature extraction network also shares weights. The input of both networks is a complete image, and the output is a feature map of each pixel, whose length and width are the same as the input image size. The backbone network module of the present invention has many optional schemes, and a suitable backbone network can be selected according to demand.

[0163] In a possible implementation manner, the variable feature extraction network and the invariant feature extraction network both include 4 convolution modules.

[0164] The first convolution module includes a 7×7 convolution, a batch normalization and a ReLU layer with a stride of 2 and a padding of 3.

[0165] The second convolution module includes a 3×3 convolution, a batch normalization, a ReLU layer, followed by a 3×3 convolution, a batch normalization, and finally a residual connection between the initial input features and the features updated by the convolution, and a ReLU is used on them.

[0166] The last two convolution modules include a 3×3 convolution, a batch normalization, a ReLU layer, followed by a 3×3 convolution, a batch normalization, and then a downsampling with a stride of 2. Finally, the residual connection is used to connect the initial input features and the features updated by the convolution, and a ReLU is used on them.

[0167] In a possible implementation manner, the variable and invariant fusion module includes an element-wise multiplication, a 3×3 convolution, a batch normalization layer, and an activation function, followed by an upsampling and a maximum pooling, and finally element-wise addition with the original features to obtain the final fused features.

[0168] The image reconstruction network reconstructs and learns the input features to obtain the reconstructed image.

[0169] In one possible implementation manner, the image reconstruction network consists of four deconvolution modules, wherein the first three deconvolution modules each contain a 3×3 convolution, a batch normalization, and a ReLU layer, with a stride of 2 and a padding of 1, and the convolution kernel size in the last deconvolution module is 1×1, with a stride of 1 and a padding of 0.

[0170] The image feature matching loss function with reconstruction constraint term constrains the network to accurately restore the image block to be matched, thereby ensuring that the decoupled learning process does not lose effective information, and the separated variable feature information and invariant feature information are the image blocks to be matched with practical restoration significance. The reconstruction loss constraint term proposed in the present invention can be combined with the loss function of the current mainstream method.

[0171] In a possible implementation manner, a reconstruction constraint loss term L for the image reconstruction network is constructed by combining the basic network module selected in this embodiment and the designed learning network for reconstructing the image. recon for:

[0172]

[0173] Where i represents the i-th pixel in the image.

[0174] In particular, the reconstruction loss form here is not only the formula (13), but also the following reconstruction loss form:

[0175]

[0176] In a possible embodiment, the reconstruction constraint loss term L recon With the weight λ set recon The image matching loss function based on variable and invariant decoupling is formed by combining with the loss function of the invariant feature extraction network itself.

[0177] In the specific implementation, this embodiment selects the coarse matching loss L coarse And the exact matching loss L fine To constrain the similarity of image features, the network's constraint on coarse matching is L coarse is the negative log-likelihood loss on the confidence matrix Pc, matching the true value It is the bidirectional nearest neighbor result of two sets of 1 / 8 resolution grids.

[0178]

[0179] The exact matching loss L fine L2 loss is used for constraint.

[0180]

[0181] The overall loss function is: L total =L coarse +L fine +λ recon L recon .

[0182] Step 3: Based on the size of the overlap value of the image pairs in the Megadepth dataset, the image pairs are divided into three categories: simple, general, and difficult. Then, a certain number of image pairs are randomly selected from the three categories in a ratio of 1:1:1 to form a training dataset, and all images are further scaled to a grayscale image of 640×640. Based on the training dataset, the decoupled variables and invariants learning network is trained. The input of the network is the entire training dataset, and the output is the dense feature representation of the corresponding grayscale image, thus obtaining the trained image descriptor learning network based on variable and invariant decoupling. Training dataset Training dataset

[0183] In the network training phase, a training data set is constructed based on the MegaDepth data set to train the network of the present invention. In training, an epoch is a complete training of the model using all the images of the training data set, a batch is a small sample of the training data set that performs a back-propagation parameter update on the model weights, and iteration is the process of updating the parameters of the model using a batch of data. The present invention can select different training results, such as using the model of the last iteration as the final model, or adding a validation set during training to select the model with the best validation result as the final model, or using multiple epochs for training and then selecting the best model as the result. At the same time, batch-size also has a certain impact on the network training results, so the network hyperparameters can be adjusted according to the network training results.

[0184] In the embodiment of the present invention, a multi-epoch training method can be used, the number of epochs is 30, the batch-size is set to 4, the optimizer is adamw, and the learning rate is 6×10 -3 .

[0185] In a possible implementation manner, step 3 further includes:

[0186] Step 4, construct a test set based on the data set, that is, including but not limited to HPatches, COLMAP, MegaDepth, ScanNet and other data sets to construct a test set. Specifically, for HPatches, the first image in each scene and the subsequent five images are directly combined into 5 pairs of image pairs, with a total of 116 scenes, so there are a total of 580 pairs of image pairs. For COLMAP and MegaDepth, the entire data set can be divided into three categories: easy (0°-15°), moderate (15°-30°), and hard (30°-60°) according to the angle difference between the image pairs. For ScanNet, the entire data set can be divided into three categories: easy (10 frames), moderate (30 frames), and hard (60 frames) according to the frame number difference between the image pairs. Based on the test set, the trained image matching network based on variable and invariant decoupling is applied to the real scene for testing to obtain matching results; the image matching results are evaluated by multiple indicators using the evaluation method.

[0187] In the specific implementation, real scenes such as 3D reconstruction, pose estimation, image matching, etc. Use the current popular evaluation methods to evaluate multiple indicators, such as homography matrix estimation, relative pose estimation accuracy, etc.

[0188] In a possible embodiment, the evaluation method is used to evaluate the performance of the descriptor using multiple indicators, including: using homography matrix estimation accuracy to evaluate the performance in downstream tasks of planar scenes. Referring to SuperPoint, the correctness of the homography matrix evaluation between each pair of images is defined by the following formula:

[0189]

[0190] Among them, e represents the given pixel threshold, and x i They are the four corner points of the source image transformed by the algorithm estimation and the true homography matrix respectively.

[0191] In a possible embodiment, the evaluation method is used to evaluate the descriptor performance using multiple indicators, including: using relative posture estimation accuracy (Rotation Estimation Accuracy and Translation Estimation Accuracy) to evaluate the performance in downstream tasks of non-planar scenes, and calculating the angle error based on the rotation matrix and the actual rotation matrix. The calculation rule is:

[0192]

[0193] in, Represents the angle prediction value, R gt Indicates the true value of the angle.

[0194] When the angle θ is less than a given threshold, the estimation is considered to be wrong. The translation estimation accuracy is also calculated using the angle error, because the existing algorithm can only calculate the translation vector of unit size.

[0195] As an optional embodiment, the present invention conducts a verification experiment based on an end-to-end image matching learning network that has performed well in recent years, and gives the following comparison results on the public image pair datasets MegaDepth, COLMAP, and HPatches.

[0196] Table 1 shows the relative pose estimation results of learning-based and traditional manual descriptor methods at MegaDepth in recent years. The data with the highest accuracy is marked in bold, and the second highest is underlined.

[0197] Table 1

[0198]

[0199] Table 2

[0200]

[0201] Table 2 shows the relative pose estimation results of learning-based and traditional manual descriptor methods on COLMAP in recent years. The data with the highest accuracy is marked in bold, and the second highest is underlined.

[0202] Table 3

[0203]

[0204] Table 3 is a comparison of the homography matrix estimation accuracy of learning-based and traditional manual descriptor methods on HPatches at different pixel thresholds in recent years. The data with the highest accuracy is marked in bold, and the second highest is underlined.

[0205] It can be seen from the results in the above table that the method of this embodiment is effective.

[0206] like Figure 6 As shown, the image decoupling system based on invariant features and variable features of this embodiment is mainly used to implement the image decoupling method based on invariant features and variable features of the above embodiment, including:

[0207] An image decoupling network construction module is used to construct an image decoupling network based on invariant features and variable features, which includes a variable feature extraction network and an invariant feature extraction network;

[0208] A training module is used to select any two image blocks to be matched in a training data set, and input them into a variable feature extraction network and an invariant feature extraction network to obtain variable feature data and invariant feature data respectively; and through a pre-constructed first constraint condition, a distance between the invariant feature data output by the invariant feature extraction network reaches a first preset minimum value;

[0209] A reconstruction module, used for fusing and reconstructing the variable feature data and the invariant feature data obtained from the same image block to be matched to obtain a reconstructed image; and making the distance between the reconstructed image and the image block to be matched reach a second preset minimum value through a pre-constructed second constraint condition;

[0210] The iterative module is used to obtain a trained image decoupling network based on invariant features and variable features through iterative training, so as to be used for decoupling the invariant features and variable features of the image.

[0211] Each module is mainly used to implement each step in the image decoupling method based on invariant features and variable features, which will not be described here.

[0212] like Figure 7 As shown, the decoupled image matching method based on invariant features and variable features in an embodiment of the present invention includes the following steps:

[0213] S701, extracting all key points from a plurality of images to be tested, extracting image blocks of a certain size with the key points as the center and performing normalization processing;

[0214] S702, inputting the normalized image blocks into a pre-trained image decoupling network based on invariant features and variable features, respectively, to extract invariant feature data of each image block; the image decoupling network based on invariant features and variable features is obtained according to the image decoupling method based on invariant features and variable features in the above embodiment;

[0215] S703, finding two image blocks with the closest distance according to the extracted invariant feature data as a pair of matching image blocks;

[0216] S704: Complete the matching of the image to be tested according to the multiple pairs of matching image blocks.

[0217] like Figure 8 As shown, the present invention also provides an image matching system based on decoupling of invariant features and variable features, which is used to implement the above method embodiment, and the system includes:

[0218] A key point extraction module is used to extract all key points in a pair of images to be matched, extract image blocks of a certain size with the key points as the center, and perform normalization processing;

[0219] Image decoupling, for inputting the normalized image blocks into a pre-trained image decoupling network based on invariant features and variable features, respectively, to extract the invariant feature data of each image block as the invariant feature of the corresponding key point; the image decoupling network based on invariant features and variable features is obtained according to the image decoupling method based on invariant features and variable features according to claim 1;

[0220] The matching module is used to complete the matching of all key points on the image pair to be matched according to the invariant features of each key point.

[0221] The present application also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, and when the program is executed by a processor, a corresponding function is realized. When the computer-readable storage medium of this embodiment is executed by a processor, the image decoupling and matching method based on invariant features and variable features of the method embodiment is realized.

[0222] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0223] The order of execution of each step in the above embodiment does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0224] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. An image decoupling method based on invariant features and variable features, characterized in that: The following steps are involved: Constructing an image decoupling network based on invariant features and variable features, which includes a variable feature extraction network and an invariant feature extraction network; Select any two image blocks to be matched in the training data set, input them into the variable feature extraction network and the invariant feature extraction network respectively, and decouple them to obtain variable feature data and invariant feature data; The distance between the invariant feature data output by the invariant feature extraction network reaches a first preset minimum value through a pre-constructed first constraint condition; The variable feature data and the invariant feature data obtained by decoupling the same image block to be matched are fused and reconstructed to obtain a reconstructed image; a second constraint condition constructed in advance is used to make the distance between the reconstructed image block and the image block to be matched reach a second preset minimum value; Through iterative training, a trained image decoupling network based on invariant features and variable features is obtained to be used for decoupling the invariant features and variable features of the image.

2. The image decoupling method based on invariant features and variable features according to claim 1, characterized in that: The first constraint condition is the triplet loss function determined by the HardNet algorithm; the second constraint condition is the reconstruction constraint loss function constructed according to the relationship between the reconstructed image block and the image block to be matched.

3. The image decoupling method based on invariant features and variable features according to claim 1, characterized in that: Depending on the training data set, the image block to be matched is a local image block cropped with the key point as the center or a complete image of the original size.

4. The image decoupling method based on invariant features and variable features according to claim 3 is characterized in that: Based on the Brown dataset, a certain number of image block pairs are randomly selected to form the initial dataset. For each pair of image blocks, a most similar image block is selected from other image pairs to form a triplet. The constructed multiple triplets are then used as the training dataset for the image decoupling network. The training goal of each training is to minimize the average value of the triplet loss.

5. The image decoupling method based on invariant features and variable features according to claim 1, characterized in that: Also includes the steps: The image decoupling network is tested using a test set, and a preset evaluation method is used to perform multi-index evaluation on the descriptors of the variable feature data and the invariant feature data output by the image decoupling network.

6. The image decoupling method based on invariant features and variable features according to claim 5, characterized in that: The test set is specifically classified according to the difference in shooting angles between the image pairs in the dataset or classified according to the difference in baseline width between the image pairs before testing.

7. The image decoupling method based on invariant features and variable features according to claim 1, characterized in that: The reconstructed image block is specifically reconstructed through an image reconstruction network, which includes multiple deconvolution modules.

8. An image decoupling system based on invariant features and variable features, characterized in that: include: An image decoupling network construction module is used to construct an image decoupling network based on invariant features and variable features, which includes a variable feature extraction network and an invariant feature extraction network; A training module is used to select any two image blocks to be matched in a training data set, and input them into a variable feature extraction network and an invariant feature extraction network respectively to decouple the variable feature data and the invariant feature data; and through a pre-constructed first constraint condition, the distance between the invariant feature data output by the invariant feature extraction network reaches a first preset minimum value; A reconstruction module, used for fusing and reconstructing the variable feature data and the invariant feature data obtained from the same image block to be matched to obtain a reconstructed image; and making the distance between the reconstructed image block and the image block to be matched reach a second preset minimum value through a pre-constructed second constraint condition; The iterative module is used to obtain a trained image decoupling network based on invariant features and variable features through iterative training, so as to be used for decoupling the invariant features and variable features of the image.

9. An image matching method based on decoupling of invariant features and variable features, characterized in that: The following steps are involved: For any pair of images to be matched, extract all key points in the image, extract image blocks of a certain size with the key points as the center and perform normalization processing; The normalized image blocks are respectively input into a pre-trained image decoupling network based on invariant features and variable features to extract invariant feature data of each image block; the image decoupling network based on invariant features and variable features is obtained according to the image decoupling method based on invariant features and variable features according to claim 1; Find the two image blocks with the closest distance according to the extracted invariant feature data as a pair of matching image blocks; The matching of the image to be tested is completed based on multiple pairs of matching image blocks.

10. An image matching system based on decoupling of invariant features and variable features, characterized in that: include: A key point extraction module is used to extract all key points in a pair of images to be matched, extract image blocks of a certain size with the key points as the center, and perform normalization processing; Image decoupling, for inputting the normalized image blocks into a pre-trained image decoupling network based on invariant features and variable features, respectively, to extract the invariant feature data of each image block as the invariant feature of the corresponding key point; the image decoupling network based on invariant features and variable features is obtained according to the image decoupling method based on invariant features and variable features according to claim 1; The matching module is used to complete the matching of all key points on the image pair to be matched according to the invariant features of each key point.

Images