A method for registering heterogeneous images based on domain invariance

Abstract

The application discloses a kind of based on field invariability heterologous image registration method, and source field image sample is extracted after the feature of source field encoder and is scaled, and source field encoder, key point detection network and descriptor extraction network are trained;Fixed source field encoder, key point detection network and descriptor extraction network, use generative adversarial network to train target field encoder, so that the feature output by target field and source field is constrained in the same space;Image of source field and target field is respectively input into trained source field encoder and target field encoder, and the feature output is obtained key point and descriptor after key point detection network and descriptor extraction network, then similar descriptor is found by similarity regularization, to find the key point pair registered by calculating descriptor distance.The application solves the problem that key point matching is not accurate enough caused by feature distribution inconsistency when image registration is carried out on images with different distributions by generative adversarial network and similarity regularization.

Description

Technical Field

[0001] This invention relates to an image registration method, and more particularly to a heterogeneous image registration method based on domain invariance. Background Technology

[0002] Image registration is the process of matching two or more images acquired at different times, by different sensors (imaging devices), or under different conditions (weather, illumination, camera position and angle, etc.) to establish a one-to-one correspondence between different positions in the images.

[0003] Existing technical solutions include:

[0004] 1. Traditional feature-based image registration methods mainly include four steps: feature extraction, feature matching, model transformation parameter estimation, and image resampling. Examples include feature extraction methods based on Scale Invariant Feature Transform (SIFT) and Speeded-Up Robust Features (SURF).

[0005] 2. Deep learning-based image registration methods mainly include the LF-Net network proposed by Ono et al., which consists of two parts: a detector network composed of fully convolutional networks and a feature descriptor network. The detector is used to obtain the location, scale, and orientation of keypoints, while the feature descriptor network is used to generate feature descriptors. End-to-end training further improves the matching effect. Zhao et al. used hypercolumn technology to integrate multi-scale features from CNNs to match common keypoints in infrared and visible light images.

[0006] Most of the methods mentioned above cannot solve the problem of heterogeneous image registration. That is, when the image distributions differ significantly, the accuracy of keypoint matching drops drastically. Existing hyperarray-based infrared and visible light image registration methods, while extracting common features from heterogeneous images, still cannot properly align features from different distributions within the same distribution space by relying solely on multi-scale features. Furthermore, these methods require a large number of visible light and infrared images with roughly the same image content, which is difficult to collect in practical applications due to the challenge of obtaining a large number of images with one-to-one matching. Summary of the Invention

[0007] The purpose of this invention is to provide a heterogeneous image registration method based on domain invariance. By using generative adversarial networks and similarity regularization, this method solves the problem of inaccurate key point matching caused by inconsistent feature distributions when registering images with different distributions.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] A heterogeneous image registration method based on domain invariance includes the following stages:

[0010] In the preprocessing stage, a large number of affine transformations are performed on the source neighborhood image samples to obtain the transformed positions of the same image key points;

[0011] The training phase includes the following steps:

[0012] Step 1: After feature extraction and scaling of a large number of source neighborhood image samples by the source neighborhood encoder, the source neighborhood encoder, key point detection network, and descriptor extraction network are trained.

[0013] Step 2: Fix the source domain encoder, key point detection network, and descriptor extraction network, and train the target domain encoder using generative adversarial network and similarity regularization, so that the features output by the target domain encoder and the source domain encoder are constrained to the same space;

[0014] During the application phase, the source and target domain images are input into the trained source and target domain encoders, respectively. The features output by the source and target domain encoders are processed by a shared keypoint detection network and descriptor extraction network to obtain keypoints and descriptors. Then, L2 regularization is used to find similar descriptors, and the descriptor distance is calculated to find the registered keypoint pairs.

[0015] Preferably, step 1 specifically includes the following steps:

[0016] Step 1-1: The source neighborhood image undergoes feature extraction and scaling via a source neighborhood encoder to obtain half of the original image. n Scale characteristics. Where n is the number of downsampling operations performed by the source neighborhood encoder;

[0017] Steps 1-2: The keypoint detection network consists of a 3x3, 64-channel convolutional layer followed by a sigmoid layer. (Original image 1 / 2) n The scale features are obtained by convolution to obtain features of depth 64, where the vector of each pixel represents the probability of whether each point in the 8x8 region of the original image includes a key point. After the vector matrix is ​​transformed, it is passed through sigmoid to obtain the confidence of whether each pixel is a key point.

[0018] 1-3. The descriptor extraction network consists of a 3x3, 8-channel convolution layer followed by an interpolation layer, which reduces the original image by 1 / 2. n The scale is convolved to obtain a descriptor with a dimension of 8, and then the feature size of the original image is obtained by interpolation and magnification.

[0019] Preferably, the keypoint detection network is trained using a cross-entropy loss function, where the labeled keypoints are obtained using SIFT. The loss function for the keypoint detection network is:

[0020]

[0021] Among them, y ij The label indicating whether it is a feature point, x ij This is the output of the feature point detection network, where i and j represent the coordinates.

[0022] Preferably, the descriptor extraction network obtains feature point pairs based on the transformed positions of the same image key points obtained in the preprocessing stage, and the descriptor matching loss function of the feature point pairs is measured by the L2 distance of the normalized descriptors.

[0023] Preferably, in step 2, the similarity between the features output by the source domain encoder and the target domain encoder is calculated using similarity regularization methods such as L2 and covariance. The generative adversarial network then uses this similarity to train the target domain encoder. The similarity calculation includes the following aspects:

[0024]

[0025] Constrain point-to-point consistency of features in two networks; where h s h represents the feature representation output by the source-domain encoder. t This represents the feature representation output by the target domain encoder, where k represents the feature dimension.

[0026]

[0027]

[0028] Where G represents the covariance matrix, which is used to extract correlation information between features;

[0029] A discriminator is added after the source domain encoder and the target domain encoder. This discriminator is used to determine whether the input features come from the target domain encoder or the source domain encoder. Based on the discrimination result, the training result of the target domain encoder is judged.

[0030] Preferably, the discriminator includes one convolutional layer and two fully connected layers. In the generative adversarial network, when training the discriminator, the parameters of the target domain encoder and the source domain encoder are fixed, and the discriminator is optimized by using the sigmoid loss function. When training the target domain encoder, the discriminator and the source domain encoder are fixed, and the labels of the target domain are flipped to optimize the target domain encoder so that the features extracted by the target domain encoder are similar to the features of the source domain.

[0031] The loss function of the discriminator is:

[0032]

[0033] When training the target domain encoder using a generative adversarial network, the loss function is:

[0034]

[0035] Where D(x) represents the output of the discriminator.

[0036] The beneficial effects of this invention are as follows:

[0037] 1. To address the problem that traditional methods cannot handle images from different sources with different feature distributions.

[0038] 2. For the problem that there are few images in the target domain and it is difficult to collect and label key point pairs in reality for matching heterogeneous source images (such as matching infrared and visible light images), this method can create image pairs without any matching images, simply by using affine transformation in a single source.

[0039] 3. By using similarity regularization and generative adversarial networks, where the similarity metric introduces covariance distance calculation to constrain the consistency of feature vector relevance between points, the target domain features can be mapped to the source domain distribution, thus solving the problem of inconsistent distribution. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of a heterogeneous image registration method based on domain invariance.

[0041] Figure 2 This is a schematic diagram of the target domain encoder and the source domain encoder. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0043] This embodiment illustrates a heterogeneous image registration method based on domain invariance. By employing a generative adversarial network (GAN) and similarity regularization, it maps the target domain image to the distribution of source domain images, enabling the network to be effectively applied to heterogeneous images. See also... Figure 1 As shown, it includes the following stages:

[0044] In the preprocessing stage, a large number of affine transformations are performed on the source neighborhood image samples to obtain the transformed positions of the same image key points, which can then be used to create a large number of registered image pairs.

[0045] The training phase includes the following steps:

[0046] 1. After feature extraction and scaling of a large number of source neighborhood image samples by the source neighborhood encoder, the source neighborhood encoder, keypoint detection network, and descriptor extraction network are trained. This specifically includes the following steps:

[0047] 1-1. The source neighborhood image undergoes feature extraction and scaling via a source neighborhood encoder to obtain half of the original image. n The scale characteristics are defined as follows: Here, n represents the number of downsampling operations performed by the source-neighborhood encoder. Too many downsampling operations result in a large receptive field, leading to a loss of detail information, such as keypoints. Conversely, too few downsampling operations reduce the receptive field, and the feature description of keypoints often depends on their surrounding information. This embodiment uses three downsampling operations as an example, based on experimental results. After feature extraction and scaling by the source-neighborhood encoder, the source-neighborhood image yields features at 1 / 8 the scale of the original image.

[0048] 1-2. The keypoint detection network consists of a 3x3, 64-channel convolutional layer and a sigmoid layer. The features of the original image at a scale of 1 / 8 are converted into features of depth 64 through convolution. The vector of each pixel represents the probability of whether each point in the original image contains a keypoint within an 8x8 region. After the vector matrix is ​​transformed, the sigmoid function is used to obtain the confidence score of whether each pixel is a keypoint.

[0049] The keypoint detection network can be trained using the cross-entropy loss function, where the labeled keypoints are obtained using SIFT. The loss function for the keypoint detection network is:

[0050]

[0051] Where y ij The label indicating whether it is a feature point, x ij This is the output of the feature point detection network, where i and j represent the coordinates.

[0052] 1-3. The descriptor extraction network consists of a 3x3, 8-channel convolution layer followed by an interpolation layer. The original image at 1 / 8 scale is convolved to obtain features of dimension 8, i.e., the descriptor dimension. Then, through interpolation, the image is enlarged to obtain features of the original image size.

[0053] The descriptor extraction network obtains feature point pairs based on the transformed positions of key points in the same image obtained in the preprocessing stage. The descriptor matching loss function for the feature point pairs is measured by the L2 distance of the normalized descriptors.

[0054] 2. A fixed-source domain encoder, a keypoint detection network, and a descriptor extraction network are used to train the target domain encoder using a generative adversarial network.

[0055] Because there are few images in the target domain, training deeper task modules alone is prone to overfitting. By using generative adversarial networks, L2 regularization between feature vectors, and covariance matrix regularization to train the target domain encoder, the output features of the target domain encoder are constrained to the same space as the source domain encoder.

[0056] Generative Adversarial Networks (GANs) are a method of unsupervised learning that learns by having two neural networks play against each other. This method was proposed by Ian Goodfellow et al. in 2014. A GAN consists of a generator network and a discriminator network.

[0057] Considering that heterogeneous images often share similar features in deep semantic information, this method aims to adapt the target domain encoder to images from a different distribution by adjusting the parameters of the target domain encoder during network training. Specifically, the source and target domain networks share the keypoint detection and descriptor extraction networks, but during downsampling, the source and target domain encoders do not share all parameters, such as the convolutional kernels of all convolutional layers. See [link to relevant documentation]. Figure 2 As shown, the source domain encoder and the target domain encoder have the same structure. The target domain encoder is initialized using the parameters of the source domain encoder. Through similarity regularization and generative adversarial network, the features output by the source domain encoder and the target domain encoder are mapped to the same feature space.

[0058] The similarity between the features output by the source and target domain encoders is calculated using similarity regularization. The generative adversarial network then uses this similarity to train the target domain encoder. The similarity calculation includes the following aspects:

[0059]

[0060]

[0061] Direct L2 regularization is used to constrain the point-to-point consistency of features between two networks; where h s h represents the feature representation output by the source-domain encoder. t This represents the feature representation output by the target domain encoder, where k represents the feature dimension.

[0062]

[0063] Here, G represents the covariance matrix. The covariance matrix can extract the correlation information between features; that is, in addition to the shallow structure, the correlation between feature points must also be consistent.

[0064] On the other hand, a discriminator is connected after the source and target domain encoders. This discriminator determines whether the input features come from the target or source domain encoders, and evaluates the training results of the target domain encoder based on the discrimination result. The discriminator consists of one convolutional layer and two fully connected layers. In generative adversarial networks, when training the discriminator, the parameters of the target and source domain encoders are fixed, and the discriminator is optimized using a sigmoid loss function. When training the target domain encoder, the discriminator and source domain encoders are fixed, and the target domain labels are flipped to optimize the target domain encoder so that its extracted features are similar to the source domain features.

[0065] The loss function of the discriminator is:

[0066]

[0067] The loss function for adversarial training of the target domain encoder is:

[0068]

[0069] Where D(x) represents the output of the discriminator.

[0070] During the network testing and application phase, the source and target domain images are input into the trained source and target domain encoders, respectively. The features output by the source and target domain encoders are processed by a shared keypoint detection network and descriptor extraction network to obtain keypoints and descriptors. Then, L2 regularization is used to find similar descriptors, and the descriptor distance is calculated to find the registered keypoint pair.

[0071] The above method can be used not only for image registration between source and target domains, but also for image registration between target domains and between source domains.

[0072] It is understood that those skilled in the art can make equivalent substitutions or modifications to the technical solution and inventive concept of the present invention, and all such substitutions or modifications should fall within the protection scope of the appended claims.

Claims

1. A heterogeneous image registration method based on domain invariance, characterized in that... It includes the following stages: In the preprocessing stage, a large number of affine transformations are performed on the source neighborhood image samples to obtain the transformed positions of the same image key points; The training phase includes the following steps: Step 1: After feature extraction and scaling of a large number of source neighborhood image samples by the source neighborhood encoder, the source neighborhood encoder, key point detection network, and descriptor extraction network are trained. Step 2: Fix the source domain encoder, key point detection network, and descriptor extraction network. Calculate the similarity between the features output by the source domain encoder and the target domain encoder using similarity regularization. Generative adversarial network then uses this similarity to train the target domain encoder. Similarity calculation includes the following aspects: Constrain point-to-point consistency of features between two networks; where h s h represents the feature representation output by the source-domain encoder. t This represents the feature representation output by the target domain encoder, where k represents the feature dimension. Where G represents the covariance matrix, which is used to extract correlation information between features; A discriminator is added after the source domain encoder and the target domain encoder. This discriminator is used to determine whether the input features come from the target domain encoder or the source domain encoder. Based on the discrimination result, the training result of the target domain encoder is judged. During the application phase, the source and target domain images are input into the trained source and target domain encoders, respectively. The features output by the source and target domain encoders are processed by a shared keypoint detection network and descriptor extraction network to obtain keypoints and descriptors. Then, L2 regularization is used to find similar descriptors, and the descriptor distance is calculated to find the registered keypoint pairs.

2. The heterogeneous image registration method based on domain invariance according to claim 1, characterized in that... Step 1 specifically includes the following steps: Step 1-1: The source neighborhood image undergoes feature extraction and scaling via a source neighborhood encoder to obtain half of the original image. n The scale characteristics; where n is the number of downsampling operations included in the source neighborhood encoder; Steps 1-2: The keypoint detection network consists of a 3x3, 64-channel convolutional layer followed by a sigmoid layer. (Original image 1 / 2) n The scale features are obtained by convolution to obtain features of depth 64, where the vector of each pixel represents the probability of whether each point in the 8x8 region of the original image includes a key point. After the vector matrix is ​​transformed, it is passed through sigmoid to obtain the confidence of whether each pixel is a key point. 1-3. The descriptor extraction network consists of a 3x3, 8-channel convolution layer followed by an interpolation layer, which reduces the original image by 1 / 2. n The scale is convolved to obtain a descriptor with a dimension of 8, and then the feature size of the original image is obtained by interpolation and magnification.

3. The heterogeneous image registration method based on domain invariance according to claim 2, characterized in that... The keypoint detection network is trained using the cross-entropy loss function, where the labeled keypoints are obtained through SIFT. The loss function of the keypoint detection network is: in, A label indicating whether it is a feature point. This is the output of the feature point detection network, where i and j represent the coordinates.

4. The heterogeneous image registration method based on domain invariance according to claim 2, characterized in that... The descriptor extraction network obtains feature point pairs based on the transformed positions of key points in the same image obtained in the preprocessing stage. The descriptor matching loss function for the feature point pairs is measured by the L2 distance of the normalized descriptors.

5. The heterogeneous image registration method based on domain invariance according to claim 1, characterized in that... The discriminator consists of one convolutional layer and two fully connected layers. In generative adversarial networks, when training the discriminator, the parameters of the target domain encoder and the source domain encoder are fixed, and the discriminator is optimized using the sigmoid loss function. When training the target domain encoder, the discriminator and the source domain encoder are fixed, and the labels of the target domain are flipped to optimize the target domain encoder so that the features extracted by the target domain encoder are similar to the features of the source domain. The loss function of the discriminator is: When training the target domain encoder using a generative adversarial network, the loss function is: Where D() represents the discriminator output.

Images