A heterogeneous image template matching method based on feature adversarial network

Abstract

The present invention discloses a heterogeneous image template matching method based on a feature adversarial network, comprising the following steps: constructing a training set; constructing a feature extraction network, a feature mapping layer and a modal discrimination network respectively; using the training data, using a generative adversarial strategy to iteratively train the network, and matching the heterogeneous images of the matching data set. The present invention is based on adversarial learning, and achieves a dynamic Nash equilibrium by performing a maximum-minimum game through a feature mapper and a modal discrimination network. In the feature mapper, in addition to calculating the structural loss of the mapping features of the two modalities, texture complexity calculation and matching result constraints are introduced to ensure that the cross-modal features still have good representation capabilities after being projected into a common subspace, thereby indirectly realizing the homologous transformation of heterogeneous images, reducing the difficulty of matching, and improving the accuracy of heterogeneous image matching.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a heterogeneous image template matching method based on a feature adversarial network. Background Art

[0002] Image registration technology is the basic part of image processing. Heterogeneous image registration technology can make up for the shortcomings of single sensor images in image registration technology and is a key technology studied in the current image processing field. There are differences in the imaging mechanisms of optical images and SAR satellite images. Realizing accurate registration of optical and SAR satellite images has become a difficulty and key issue in optical image SAR satellite image registration. Among multi-source image data, SAR and optical images are the most typical: optical images conform to the visual characteristics of the human eye and are easy to identify, while SAR satellite images have the characteristics of all-day, all-weather, and high resolution. Therefore, the effective fusion of the complementary information of SAR and optical images for the same object helps to detect target characteristics more comprehensively and objectively.

[0003] With the success of deep learning technology in the field of image processing, neural network convolution operators can obtain more complex and robust features than artificial operators, and have the potential to extract common features in scenes with strong illumination changes and strong radiation differences between SAR and optical images. At present, the innovations of heterogeneous image matching methods are mostly centered around feature extraction networks, and different strategies and structures are used to improve the network's extraction capabilities and adaptability to different objects. Due to the different imaging principles of heterogeneous images, there are large differences between similar objects in different modal images, and the matching results obtained by using existing matching methods have low accuracy. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a heterogeneous image template matching method based on a feature adversarial network. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0005] A heterogeneous image template matching method based on a feature adversarial network comprises the following steps:

[0006] Step 10, constructing a training set; the training set includes image pairs consisting of multiple registered SAR images and visible light images, SAR image templates, and corresponding matching labels; wherein the SAR image template is an image area of ​​a preset size with any pixel point of the SAR image registered with the visible light image as the upper left corner coordinate, and the matching label is the upper left corner pixel coordinate of the SAR image template;

[0007] Step 20, constructing a feature extraction network, a feature mapping layer and a modal discrimination network respectively; wherein the feature mapping layer is a three-layer feedforward neural network with tanh as an activation function, and the modal discrimination network includes three fully connected layers and a Softmax layer;

[0008] Step 30, inputting the image pairs of the training set into the feature extraction network and the feature mapping layer in sequence to obtain a sequence of SAR mapping feature maps and a sequence of visible light mapping feature maps;

[0009] Step 40, calculating its L according to the sequence of the SAR mapping feature graphs, the sequence of the visible light mapping feature graphs, the SAR image template and the modal discrimination network 2 norm, entropy of gray-level co-occurrence matrix, matching loss and discriminant loss;

[0010] Step 50, according to the L 2 norm, the entropy of the gray-level co-occurrence matrix, the matching loss and the regularization function to determine the total mapping loss;

[0011] Step 60, updating the parameters of the feature mapping layer and the modal discrimination network using the Adam algorithm according to the total mapping loss and the discrimination loss to obtain an updated feature mapping layer and an updated modal discrimination network;

[0012] Step 70: determining the matching result of the SAR image and the visible light image according to the data set to be matched, the feature extraction network and the updated feature mapping layer.

[0013] In one embodiment of the present invention, the feature extraction network includes a first feature extraction subnetwork and a second feature extraction subnetwork with the same structure, and the network weights of the first feature extraction subnetwork and the network weights of the second feature extraction subnetwork are not shared;

[0014] The first feature extraction subnetwork is used to extract features from the SAR image and output a SAR image feature map sequence;

[0015] The second feature extraction subnetwork is used to extract features from the visible light image and output a visible light image feature map sequence;

[0016] The feature mapping layer includes a first mapping subnetwork and a second mapping subnetwork with the same structure;

[0017] The first mapping subnetwork is used to map the SAR image feature map sequence and output a SAR mapping feature map sequence;

[0018] The second mapping subnetwork is used to map the visible light image feature map sequence and output a visible light mapping feature map sequence.

[0019] In one embodiment of the present invention, step 30 includes:

[0020] Step 31: input the SAR image and the corresponding visible light image of the image pair of the training set into the first feature extraction subnetwork and the second feature extraction subnetwork respectively, and output SAR image feature map sequences respectively. And the visible light image feature map sequence Among them, s i represents the feature map of the i-th SAR image in the training set, o i represents the feature map of the i-th visible light image in the training set, and n represents the number of image pairs consisting of SAR images and visible light images;

[0021] Step 32: input the SAR image feature map sequence and the visible light image feature map sequence into the first mapping subnetwork and the second mapping subnetwork respectively, and output a sequence of SAR mapping feature maps. and a sequence of visible light mapping feature maps where f S and f O denote the mapping function of the first mapping sub-network and the mapping function of the second mapping sub-network respectively, θ S represents the network parameters of the first mapping subnetwork, θ O The network parameters of the second mapping sub-network are represented.

[0022] In one embodiment of the present invention, step 40 includes:

[0023] Step 41: The sequence SP of the SAR mapping feature graph S and the image sequence SP of the visible light mapping feature O The modal discrimination network is input for classification, and the discrimination loss is calculated; the loss function of the discrimination loss is:

[0024]

[0025] Where D represents the discriminator of the modality discrimination network, θ D Represents the network parameters of the mode discrimination network;

[0026] Step 42, in a feature mapper, mapping the sequence SP of the feature map according to the SAR S and the sequence SP of the visible light mapping feature map O Calculate SP S and SP O L 2 Norm:

[0027] L 2=||SP s -SP o || 2 ;

[0028] Step 43: Calculate the entropy L of the total gray-level co-occurrence matrix in the feature mapper according to the sequence of the SAR mapping feature graphs and the sequence of the visible light mapping feature graphs. Ent :

[0029]

[0030] Among them, f Ent Represents the mapping function of the mean entropy of all feature channel images of a feature map to their corresponding gray-level co-occurrence matrix.

[0031] Step 44, in the feature mapper, the matching loss is calculated according to the SAR image mapping feature map, the visible light mapping feature map and the matching label: the Euclidean distance between the SAR image template area corresponding to the SAR image mapping feature map and the visible light mapping feature map is calculated by sliding window as a similarity index, and the upper left corner point of the SAR image template corresponding to the minimum Euclidean distance is taken as the matching point g i ', calculate g i 'Image matching label g corresponding to the SAR image template i The loss is also the matching loss, and the loss function of the matching loss is:

[0032]

[0033] Among them, MSE is the mean square error loss function.

[0034] In one embodiment of the present invention, the total mapping loss is:

[0035]

[0036] Among them, α and β are hyperparameters, L r represents the regular function, represents the parameters of the first layer of the first mapping subnetwork, N Layer Indicates the number of network layers of the mapping subnetwork. Represents the parameters of the lth layer of the second mapping subnetwork.

[0037] In one embodiment of the present invention, step 60 includes:

[0038] Step 61, constructing an objective function according to the total mapping loss and the discrimination loss; wherein the expression of the objective function is:

[0039]

[0040] in, represents the updated network parameters of the first mapping subnetwork, represents the updated network parameters of the second mapping subnetwork, Represents the network parameters of the updated modal discrimination network;

[0041] Step 62: According to the objective function, the Adam algorithm is used to update the parameters of the feature mapping layer and the modal discrimination network to obtain an updated feature mapping layer and an updated modal discrimination network.

[0042] In one embodiment of the present invention, step 70 includes: inputting the SAR image and the visible light image in the to-be-matched data set into the feature extraction network and the updated feature mapping layer in sequence to obtain a sequence of SAR target mapping feature maps and a sequence of visible light target mapping feature maps, and then calculating the Euclidean distance between the SAR target mapping feature map and the visible light target mapping feature map by means of a sliding window, and obtaining the coordinates of the upper left corner pixel point of the SAR target mapping feature map corresponding to the minimum Euclidean distance in the visible light target mapping feature map.

[0043] Beneficial effects of the present invention:

[0044] Based on adversarial learning, the present invention achieves a dynamic Nash equilibrium by performing a maximum-minimum game through a feature mapper and a modal discrimination network. In the feature mapper, in addition to calculating the structural loss of the mapping features of the two modalities, texture complexity calculation and matching result constraints are introduced to ensure that the cross-modal features still have good representation capabilities after being projected into the common subspace, thereby indirectly realizing the homologous transformation of heterogeneous images, reducing the difficulty of matching, and improving the accuracy of heterogeneous image matching.

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a heterogeneous image template matching method based on a feature adversarial network provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the network structure of a heterogeneous image template matching method based on a feature adversarial network provided by an embodiment of the present invention;

[0048] Figure 3a The matching experimental results of the existing matching method of SAR images and visible light images of urban areas in the OSdataset dataset and the matching experimental results of the method of this embodiment;

[0049] Figure 3bThe matching experimental results of the existing matching method of SAR images and visible light images of rural areas in the OSdataset dataset and the matching experimental results of the method of this embodiment;

[0050] Figure 4a The matching experimental results of the existing matching method and the method of this embodiment for SAR images and visible light images of urban areas in the SEN1-2 Dataset;

[0051] Figure 4b The matching experimental results of the existing matching method and the method of this embodiment for SAR images and visible light images of rural areas in the SEN1-2 Dataset;

[0052] Figure 5 A visualization result of a pair of partial feature channels of heterogeneous images using a heterogeneous image template matching method based on a feature adversarial network provided by an embodiment of the present invention;

[0053] Figure 6 A trend graph of the value changes of the discrimination loss and the total mapping loss of a heterogeneous image template matching method based on a feature adversarial network provided in an embodiment of the present invention from the 1st epoch to the 300th epoch. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0055] The core of the method of the present invention is to find an effective common subspace that can directly compare the features of different modalities based on adversarial learning. Among them, adversarial learning includes two processes: first, the features of the two modalities extracted are feature mapped. This process aims to obtain a nonlinear function through learning to project the features of the two modalities into the common space of an intermediate modality, and obtain highly representative features that can be used for matching; second, a modality discrimination network is used to classify the generated features and accurately determine which modality the target features come from. The two processes are mixed, and effective feature mappings from different modalities to intermediate modalities are obtained through minimax games. In order to avoid the degradation of feature generation capabilities, this method further restricts the mapper and determines through matching results that the features have the discriminative power for similarity measurement.

[0056] like Figure 1 As shown, a heterogeneous image template matching method based on a feature adversarial network includes the following steps:

[0057] Step 10: construct a training set, which includes image pairs consisting of multiple registered SAR images and visible light images, SAR image templates, and corresponding matching labels.

[0058] The SAR image template is an image area of ​​a preset size with any pixel point of the SAR image registered with the visible light image as the upper left corner coordinate, and the matching label is the upper left corner pixel coordinate of the SAR image template.

[0059] Specifically, images with a size of 512×512 are selected from the open source dataset OS Dataset as the selected dataset, which contains SAR images and visible light images that have been registered;

[0060] The visible light images in the selected data set are used as search images, pixels are randomly selected as the upper left corner coordinates in each visible light SAR image, a 256×256 image is cropped as the template image, and the upper left corner coordinates are saved as the matching label of the image pair;

[0061] 80% of the image pairs in the data set are selected as the training set, and the remaining 20% ​​of the image pairs are used as the data set to be matched.

[0062] Step 20, construct a feature extraction network, a feature mapping layer and a modal discrimination network respectively; wherein the feature mapping layer is a three-layer feedforward neural network with tanh as the activation function, and the modal discrimination network includes three fully connected layers and a Softmax layer. Specifically:

[0063] A. Construct feature extraction network:

[0064] The feature extraction network includes a first feature extraction subnetwork and a second feature extraction subnetwork with the same structure, wherein the network weight of the first feature extraction subnetwork and the network weight of the second feature extraction subnetwork are not shared, that is, the network weight of the first feature extraction subnetwork and the network weight of the second feature extraction subnetwork are different;

[0065] The first feature extraction subnetwork is used to extract features from the SAR image and output a SAR image feature map sequence

[0066] The second feature extraction subnetwork is used to extract features from the visible light image and output a visible light image feature map sequence

[0067] Among them, s i represents the feature map of the i-th SAR image, o i represents the feature map of the i-th visible light image, and n represents the number of image pairs consisting of SAR images and visible light images.

[0068] The first feature extraction subnetwork and the second feature extraction subnetwork are two Attention U-Nets that do not share weights, and their outputs are feature maps with the same size as the input and 512 channels.

[0069] B. Construct feature mapping layer:

[0070] A feature mapping layer, comprising a first mapping subnetwork and a second mapping subnetwork having the same structure;

[0071] The first mapping subnetwork is used to map the SAR image features and output a sequence of SAR mapping feature maps;

[0072] The second mapping subnetwork is used to map the visible light image features and output a sequence of visible light mapping feature maps.

[0073] Among them, the first mapping sub-network and the second mapping sub-network are both three-layer feed-forward neural networks with tanh as the activation function, which are used to nonlinearly project the features of the two modes into a common subspace and reduce the number of feature channels of the two to 200.

[0074] C, build a modal discrimination network:

[0075] The Generative Adversarial Network was proposed by Ian Goodfellow. The network consists of a generator and a discriminator, which are optimized and trained through adversarial learning. The function of the generator is to convert the input noise or image into a pseudo sample and make it very close to the distribution of the real data, so that the discriminator cannot distinguish it. The function of the discriminator is to classify the generated samples and distinguish the authenticity of the generated samples, thereby guiding the optimization of the generator. The entire training process consists of two networks alternately training and updating parameters, which can be regarded as a game between the two. The training loss function is shown in the following formula:

[0076]

[0077] In the formula, x represents the real sample, z represents the noise sample, and p data represents the distribution of all real samples, p z represents the distribution of all noise samples, V(D,G) represents the cross entropy loss between the generator and the discriminator, is the expected value, D(·) is the classification probability output by the discriminator, and G(·) is the forged data output by the generator. The meaning of the loss function is that in order to obtain the optimal generator, the generated data needs to be closer to the distribution of the real data, so that the discriminator cannot distinguish the authenticity of the two. Therefore, it is necessary to maximize the discriminant loss D(G(z)); in order to obtain the optimal discriminator, that is, to be able to distinguish between real samples and forged samples to the greatest extent, to maximize its discrimination probability, it is equivalent to minimizing the discrimination probability of the generated samples. The two discriminators are trained alternately until a dynamic Nash equilibrium is reached.

[0078] Based on GAN, researchers combined CNN’s powerful feature extraction capabilities with it and implemented a variety of algorithms for generating images from images, such as DCGAN, CycleGAN, Pix2Pix, Nice-GAN, etc. The core of these algorithms is mainly reflected in two aspects. One is to use cross-layer connections in the generator network to combine shallow and deep feature encodings, supplementing shallow texture, edge and other features for deep abstract high-order features, greatly improving the quality of generated images. The second is to introduce L based on the original loss of GAN. 1 and L 2 norm, constraining the gap between the generated image and the target image so that the generated image is closer to the target image at the pixel level. With these image-to-image generation algorithms, many scholars have also tried to generate visible light images from SAR and SAR from visible light. Li et al. proposed a SAR to visible light image translation method based on CycleGAN. They added a structural similarity index (SSIM) based on CycleGAN loss, thereby improving the interpretation ability of SAR images. Zhang et al. proposed a feature-guided SAR-visible light image conversion method. They fully considered the structure and texture features of the image, performed feature matching at different layers of the generator network, and designed a loss function based on discrete cosine transform to obtain better image generation quality.

[0079] In order to verify the quality of direct cross-modal image generation, we used Pix2Pix to test the modal conversion of Opt2SAR. Direct modal conversion is very difficult, and the quality of generation is extremely unstable under different terrain environments. In images with small differences in terrain such as plains and ports, Opt2SAR can generate images close to the target image. However, for airports, due to the huge difference between the aircraft in the visible light image and the aircraft in the SAR image, it is difficult for the algorithm to learn the equivalent mapping between the two.

[0080] From the results of the preliminary experiment, it can be seen that the method of directly performing cross-modal generation to convert the two into homologous images has great limitations, and the generation results are not stable. Therefore, it is necessary to consider bidirectionally generating the two into an intermediate modality, so as to reduce the difficulty of mapping and improve the quality of conversion to the same modality.

[0081] The modal discriminator network of this embodiment is the same as the discriminator in GAN. The modal discriminator D is used to classify the modality. The mapping features from the SAR image are assigned the label [0,1], and the mapping features from the visible light image are assigned the label [1,0]. For the modal discriminator, its goal is to determine the modality of the feature as accurately as possible given a feature map from an unknown source. For the implementation of the modal discriminator network, the modal discriminator network of this embodiment uses θ with a parameter of D A three-layer feedforward neural network is constructed, and a Softmax layer is added after the last layer for classification.

[0082] After constructing the above three networks, the networks are iteratively trained using the training data and the generative adversarial strategy. Specifically:

[0083] Step 30, input the image pairs of the training set into the feature extraction network and the feature mapping layer in sequence to obtain the SAR mapping feature map and the visible light mapping feature map. Specifically, step 30 includes step 31 and step 32:

[0084] Step 31: input the SAR image and the corresponding visible light image of the image pair of the training set into the first feature extraction subnetwork and the second feature extraction subnetwork respectively, and output the SAR image feature map sequence respectively. Visible light image feature map sequence

[0085] Among them, s i represents the feature map of the i-th SAR image in the training set, o i represents the feature map of the i-th visible light image in the training set, and n represents the number of image pairs consisting of SAR images and visible light images.

[0086] Step 32: Input the SAR image feature map sequence and the visible light image feature map sequence into the first mapping subnetwork and the second mapping subnetwork respectively, and output the sequence of SAR mapping feature maps. and a sequence of visible light mapping feature maps

[0087] where f S and f O denote the mapping function of the first mapping sub-network and the mapping function of the second mapping sub-network respectively, θ S represents the network parameters of the first mapping subnetwork, θ O The network parameters of the second mapping sub-network are represented.

[0088] For example, there is an instance sequence consisting of n SAR-visible light image pairs. Since the features of the two modalities of SAR and visible light usually have different data distributions and statistical characteristics, in order to directly measure the similarity between the two, the goal is to find a common subspace SP so that the SAR feature S and the visible light feature O can be mapped to Sequence of visible light mapping feature maps The method proposed in this invention aims to learn more effective transformation features in SP for different modalities, requiring SP S and SP O It is modality-invariant and discriminable, while also ensuring the inherent cross-modal similarity structure between the two.

[0089] Step 40, calculating its L according to the sequence of SAR mapping feature maps, the sequence of visible light mapping feature maps, the SAR image template and the modal discrimination network 2 norm, entropy of gray-level co-occurrence matrix, matching loss and discrimination loss; step 40 specifically includes steps 41 to 44:

[0090] Step 41: Map the sequence SP of the SAR feature map S and the image sequence SP of the visible light mapping feature map O Input the modal discrimination network for classification and calculate the discrimination loss; the loss function of the discrimination loss is:

[0091]

[0092] Where D represents the discriminator of the modality discrimination network, θ D Represents the network parameters of the modal discrimination network. The role of the modal discrimination network is to act as an adversarial network, so the discrimination loss of this process can also be called adversarial loss, which is essentially the cross entropy loss of binary classification.

[0093] After obtaining the mapping feature map, the SAR image template, SAR mapping feature map and visible light mapping feature map are input into the feature mapper to perform correlation loss, texture complexity, matching loss and total mapping loss. The feature mapper is a module used to calculate these losses:

[0094] Step 42, in the feature mapper, the sequence SP of the feature map is mapped according to the SAR S and the sequence SP of the visible light mapping feature map O Calculate SP S and SP O L 2 Norm: L 2 =||SP s -SP o || 2 ;

[0095] After obtaining the mapping features, the goal is to minimize the gap between the features obtained for the same image pair, that is, to make SP S and SP O As similar as possible. Therefore, the most direct way is to calculate the L between the two 2 norm. So L 2 The norm can be considered as the correlation loss between the two.

[0096] Step 43, in the feature mapper, the entropy L of the total gray-level co-occurrence matrix is ​​calculated according to the sequence of SAR mapping feature maps and the sequence of visible light mapping feature maps. Ent :

[0097]

[0098] Among them, the calculation formula for the entropy of the gray level co-occurrence matrix of a single channel in a single feature map in the sequence of SAR mapping feature maps and the sequence of visible light mapping feature maps is:

[0099]

[0100] Among them, u and v represent the grayscale values ​​of two pixels in a single channel in a single feature map, respectively, and p single (u,v) represents the gray-level co-occurrence matrix of a single channel in a single feature map;

[0101] As in most existing studies, the correlation loss (L 2 The goal of the loss loss is only to make the features between the two modalities closer, and it cannot guarantee that the obtained features have representational significance. In the experiment, it was found that using only this loss as a feature mapper would cause the network's feature extraction ability to degenerate, that is, an empty feature map without any structure, texture, semantics or deep abstract information would be obtained. This happens because an empty feature map without any useful information can also satisfy the adversarial learning of the entire network, that is, the features between the two modalities are very close, and the discriminator cannot distinguish the modal source of each feature map.

[0102] In order to avoid this situation and make the mapped feature map have effective representation meanings such as structural semantics, this method adds a measure of the texture complexity of the feature, and suppresses the degradation of feature extraction ability by judging the feature texture. The specific implementation method is to calculate the entropy of the gray-level co-occurrence matrix (GLCM) of the feature map. The statistical method of the gray-level co-occurrence matrix was proposed by R. Haralick and others in the early 1970s. This method takes the pixel point with gray value i' in the image as the starting point, and counts the probability of the gray value i' at the point (dx, dy) away from the point. After traversing all the pixel points, all the statistical values ​​can be expressed as a matrix. The calculation process is: select any point (x, y) and another point (x+dx, y+dy) from the image, and set the gray value of the point pair to (u, v). Let (x, y) move on the image in a sliding window, and different (u, v) values ​​will be obtained. Let the number of gray value levels be k, then there are k combinations of (u, v). 2 The number of each (u, v) value in the entire image is counted and formed into a matrix, and the total number of (u, v) occurrences is normalized to the probability p single (u,v), the final matrix is ​​the grayscale co-occurrence matrix. When the distance (dx, dy) from the selected point is different, different joint probability matrices can be obtained. Usually, the value of (dx, dy) is selected according to the different texture characteristics of the image. For small and dense textures, small distance values ​​such as (1, 0), (1, 1), and (2, 0) are selected. When dx = 1, dy = 0, the pixel pair is in the horizontal direction, that is, the moving angle is 0 degrees; when dx = 0, dy = 1, the pixel pair is in the vertical direction, that is, the moving angle is 90 degrees; when dx = 1, dy = 1, the pixel pair is a right diagonal line, that is, the moving direction is 45 degrees to the right; when dx = -1, dy = 1, the pixel pair is a left diagonal line, that is, the moving direction is 135 degrees. By calculating the probability of the grayscale of two pixels appearing, the position coordinates can be converted into a description of the grayscale of the corresponding point, and finally the grayscale co-occurrence matrix is ​​obtained. Its mathematical expression is as follows:

[0103] p single (u,v|d,θ)={(x,y)|h single (x,y)=u,h single (x+dx,y+dy)=v;x,y=0,1,...,N-1};

[0104] Where d represents the distance between two pixels, θ represents the angle between two pixels, and h single (x,y) represents a single image and N represents N gray levels.

[0105] After the gray-level co-occurrence matrix is ​​obtained, the measure of texture complexity is obtained by calculating its entropy. According to the sequence of the SAR mapping feature map and the sequence of the visible light mapping feature map, the entropy of the total gray-level co-occurrence matrix is ​​expressed as follows:

[0106]

[0107] Among them, f Ent Represents the mapping function of all feature channel images of a feature map to the mean entropy of its corresponding gray-level co-occurrence matrix;

[0108] The entropy here represents the randomness of the amount of information contained in the image. Therefore, the entropy value indicates the complexity of the grayscale distribution of the image. The larger the value, the more complex the texture of the image.

[0109] Step 44, in the feature mapper, the matching loss is calculated according to the SAR image mapping feature map, the visible light mapping feature map and the matching label.

[0110] Specifically, in addition to texture complexity, in order to verify that the mapping features in the subspace have the ability to represent the matching problem, the Euclidean distance between the SAR image template area corresponding to the SAR image mapping feature map and the visible light mapping feature map is calculated by sliding window as the similarity index, and the upper left corner point of the SAR image template corresponding to the minimum Euclidean distance is taken as the matching point g i ', calculate g i 'Image matching label g corresponding to the SAR image template i The loss is also the matching loss, and the loss function of the matching loss is:

[0111]

[0112] Among them, MSE is the mean square error loss function.

[0113] Step 50, according to L 2 The norm, entropy of the gray-level co-occurrence matrix, matching loss, and regularization function determine the total mapping loss.

[0114] After calculating the matching loss, a regularization term is introduced to prevent the learned parameters from overfitting:

[0115]

[0116] Then, the total mapping loss is calculated in the feature mapper as:

[0117]

[0118] Among them, α and β are hyperparameters, which are used to control the proportion of the two losses in the total mapping loss. r represents the regular function, represents the parameters of the lth layer of the first mapping subnetwork, represents the parameters of the lth layer of the second mapping subnetwork, F is the F norm, N Layer represents the number of network layers of feature mapping layer, θ S represents the network parameters of the first mapping subnetwork, θ O represents all network parameters of the second mapping subnetwork. For example, α=10 and β=0.1 are set.

[0119] Step 60, using the Adam algorithm to update the parameters of the feature mapping layer and the modal discrimination network according to the total mapping loss and the discrimination loss, to obtain an updated feature mapping layer and an updated modal discrimination network. Step 60 includes:

[0120] Step 61, construct an objective function according to the total mapping loss and the discrimination loss; wherein the expression of the objective function is:

[0121]

[0122] in, represents the updated network parameters of the first mapping subnetwork, represents the updated network parameters of the second mapping subnetwork, Represents the network parameters of the updated modal discrimination network;

[0123] Step 62, according to the objective function, the Adam algorithm is used to update the parameters of the feature mapping layer and the modal discrimination network to obtain an updated feature mapping layer and an updated modal discrimination network.

[0124] Among them, the process of learning the optimal representation of the feature map is carried out by jointly minimizing the discriminant loss of the modal discriminant network and the total mapping loss. Since the optimization objectives of these two loss functions are opposite, they can be carried out simultaneously as the maximum and minimum games of GAN.

[0125] This minimax game can be implemented using a stochastic gradient descent optimization algorithm, such as the Adam algorithm. By incorporating a gradient reversal layer (GRL), the minimax problem can be optimized. The gradient reversal layer is invalid during forward propagation, and the gradient value is multiplied by -1 only during reverse propagation to achieve reversal. In this way, by adding GRL before the first layer of the modal discriminator, the minimax optimization can be performed simultaneously.

[0126] Step 70, determining the matching result of the SAR image and the visible light image according to the data set to be matched, the feature extraction network and the updated feature mapping layer: inputting the SAR image and the visible light image in the data set to be matched into the feature extraction network and the updated feature mapping layer in sequence to obtain a sequence of SAR target mapping feature maps and a sequence of visible light target mapping feature maps, and then calculating the Euclidean distance between the SAR target mapping feature map and the visible light target mapping feature map by a sliding window method, and obtaining the coordinates of the upper left corner pixel point of the SAR target mapping feature map corresponding to the minimum Euclidean distance in the visible light target mapping feature map.

[0127] A1, the SAR image and the visible light image in the to-be-matched data set are sequentially input into the feature extraction network and the updated feature mapping layer to obtain a sequence of SAR target mapping feature maps and a sequence of visible light target mapping feature maps; wherein the to-be-matched data set includes unmatched SAR image templates and visible light images. The SAR image template in the to-be-matched data set is an image region of a preset size with any pixel point of the unmatched SAR image as the upper left corner coordinate.

[0128] A2, the Euclidean distance between the SAR image template area corresponding to the SAR target image mapping feature map and the visible light target mapping feature map is calculated by sliding window as the similarity index, and the coordinates of the upper left corner pixel of the SAR image template corresponding to the minimum Euclidean distance are taken as the matching result.

[0129] Specifically, the Euclidean distance is calculated by sliding the window. This method splits the Euclidean distance calculation of the sliding window and combines it with the convolution calculation, thereby greatly reducing the calculation time. The similarity calculation for the template image with a size of W×H and the corresponding area can be expressed as follows:

[0130]

[0131] Represents the pixel with coordinates (w, h) in the SAR target mapping feature map, Represents the pixel with coordinates (w, h) in the visible light target mapping feature map.

[0132] The coordinates of the minimum value obtained after normalizing the score matrix score are the matching position, and the area of ​​a preset size with the coordinates of the minimum value as the upper left corner pixel coordinates in the visible light image is the corresponding matching position of the SAR image template in the visible light image.

[0133] The matching effect of the present invention is evaluated through specific experimental results and analysis below:

[0134] 1. The experimental environment of the experiment is Window 10 system, the CPU is Core i7-9700K, the GPU is Nvidia GeforceRTX2080Ti, and Python 3.7 is used for programming, Pytorch version 1.5.1.

[0135] 2. The datasets used in this experiment are OSdataset and SEN1-2 Dataset. The OSdataset dataset contains a total of 2673 pairs of visible light images and SAR images with a size of 512×512, and each pair is registered and rectified; the SEN1-2Dataset dataset includes 4088 pairs of visible light images and SAR images with a size of 256×256.

[0136] 3. Experimental details

[0137] like Figure 2 As shown in the figure, two non-shared weight Attention U-Nets are used in the feature extraction part to obtain a feature map with the same size as the input and 512 channels. On the feature mapping layer, a three-layer feedforward neural network with tanh as the activation function is deployed to nonlinearly project the features of the two modalities into a common subspace and reduce the number of feature channels of both to 200. For the modality classifier, three fully connected layers are used, and Softmax is added after the last layer for classification.

[0138] 4. Experimental results and conclusions

[0139] In order to compare the performance of the proposed method with other heterogeneous image matching methods, for Dataset 1, we selected grayscale, feature and depth-based methods, including NCC, NMI, CFOG, RIFT, PSiam and VSMatch. For the deep algorithm, the optimal value obtained on the local server is used as the result. Tables 1 and 2 show the experimental results.

[0140] As can be seen from Table 1, the grayscale-based methods NCC and NMI are far inferior to the feature-based and deep learning-based methods in terms of matching time, accuracy, and error. The accuracy of NCC is only 0.1083, and the result of NMI is still better than NCC, but the accuracy is only 0.2750. In the feature-based algorithm, compared with the grayscale-based algorithm, both the accuracy and error are greatly improved. Figure 3a and Figure 3bThe experimental results of a group of urban and rural areas in the OSdataset dataset are shown. It can be seen that for image pairs with strong structure and small modal differences such as urban and rural areas, various matching methods can achieve good matching results. For methods based on deep learning, the method proposed in the present invention can use a simple network to convert the features of heterogeneous images into homologous ones, thereby reducing the difficulty of matching, and the matching accuracy is also improved to 0.8583, which is significantly improved compared with previous methods, and the matching time is greatly reduced, reducing the average inference time to 2.1349 seconds. But in general, it can be seen that the method proposed in the present invention is superior to similar algorithms in terms of network inference speed and accuracy.

[0141]

[0142]

[0143] Table 1. Results of OS dataset control experiments

[0144] As can be seen from Table 2, for the SEN1-2 dataset, when the template image and the search image sizes are reduced, the matching accuracy and precision of each algorithm are significantly improved compared to the OSdatas dataset, and the reasoning speed is also faster. The similarities are that the grayscale-based methods NCC and NMI are still not as good as the feature-based and deep learning-based methods. The accuracy of NCC is only 0.1667, and the result of NMI has been greatly improved, with an accuracy of 0.5739. In the feature-based algorithm, compared with the grayscale-based algorithm, both the accuracy and the error are greatly improved. For the deep learning-based method, the method proposed in the present invention has low implementation difficulty and fast reasoning speed. The average reasoning time is reduced to 0.2369 seconds, and the matching accuracy is also increased to 0.9275, which is a significant improvement over the previous methods. Figure 4a and Figure 4b The experimental results of a set of plain areas and urban areas of the SEN1-2 Dataset are shown. It can be seen that, consistent with the previous analysis, all algorithms have good matching results on images with clear structure, strong texture and small modal differences.

[0145]

[0146] Table 2. Comparative experimental results of N1-2 dataset

[0147] But for Figure 4bIn the urban area shown, due to the imaging resolution and size, the urban area texture in the SAR image is blurred, the ground features are not obvious, and there is a large difference with the visible light. When matching, it can only be matched based on the river in the image. Therefore, on such images with high matching difficulty, the grayscale-based method has poor results, and the feature-based method also has a large error. Only the method proposed in this paper obtains relatively stable results. Therefore, it can be seen that the method proposed in this chapter is superior to similar algorithms in terms of network inference speed and accuracy.

[0148] In order to verify the matching performance of the present invention after obtaining homologous features, different similarity metrics are selected for experiments on the OSdataset data set, namely MSE, NCC, and SIFT. The results are shown in Table 3.

[0149] It can be seen from Table 3 that different similarity metrics have little impact on the final results. Since the MSE accuracy is similar and takes less time, this strategy was finally chosen. Since in this experiment, the OS dataset has images of two modalities of equal size, the mapping features of the two can be directly compared. However, in some cases, there are only small-sized SAR images as templates. Therefore, in order to cope with this situation and verify that this method is still effective in this case, the method can be adjusted a little. In the feature mapper, the matching coordinates can be used to intercept the corresponding area from the visible light to calculate L 2 loss and texture complexity, and the rest remain unchanged.

[0150]

[0151] Table 3. Matching experimental results of different similarity measures

[0152] In order to verify that the proposed method can obtain representative features in the same feature space, Figure 5 The visualization results of some feature channels of a pair of heterogeneous images are shown. It can be seen that compared with the direct cross-modal generation method, the method proposed in the present invention can effectively map the two modalities to the same intermediate modality and can retain sufficient texture features for subsequent calculations.

[0153] In the method proposed in this paper, we combine the discriminant loss and the mapping loss when optimizing the loss function. In order to explore the role of feature adversarial learning in the whole method, the values ​​of the discriminant loss and the mapping loss are sampled from the 1st epoch to the 300th epoch and displayed in Figure 6 As can be seen from the figure, during the entire training process, the mapping loss is almost monotonically decreasing and converges smoothly, while the adversarial loss first oscillates (about the first 10 epochs) and then stabilizes. Figure 6 The results in are consistent with expectations, that is, in the proposed method of the present invention, the modality discriminator serves as a directional guide for the subspace mapping process, guiding the optimization direction of the feature mapper. If the loss value increases sharply, the modality classifier cannot guide the subspace mapping process. In contrast, if the loss is optimized to zero, the modality classifier will win the minimax game, which means that the embedding layer cannot generate a modality-invariant subspace representation, making cross-modal mapping impossible.

[0154] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0155] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.

[0156] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A heterogeneous image template matching method based on feature adversarial network, characterized in that: The following steps are involved: Step 10, constructing a training set; the training set includes image pairs consisting of multiple registered SAR images and visible light images, SAR image templates, and corresponding matching labels; wherein the SAR image template is an image area of ​​a preset size with any pixel point of the SAR image registered with the visible light image as the upper left corner coordinate, and the matching label is the upper left corner pixel coordinate of the SAR image template; Step 20, constructing a feature extraction network, a feature mapping layer and a modal discrimination network respectively; wherein the feature mapping layer is a three-layer feedforward neural network with tanh as an activation function, and the modal discrimination network includes three fully connected layers and a Softmax layer; Step 30, inputting the image pairs of the training set into the feature extraction network and the feature mapping layer in sequence to obtain a sequence of SAR mapping feature maps and a sequence of visible light mapping feature maps; Step 40, calculating the L2 norm, entropy of the gray-level co-occurrence matrix, matching loss and discrimination loss according to the sequence of the SAR mapping feature graphs, the sequence of the visible light mapping feature graphs, the SAR image template and the modal discrimination network; Step 50, determining a total mapping loss according to the L2 norm, the entropy of the gray level co-occurrence matrix, the matching loss and a regularization function; Step 60, updating the parameters of the feature mapping layer and the modal discrimination network using the Adam algorithm according to the total mapping loss and the discrimination loss to obtain an updated feature mapping layer and an updated modal discrimination network; Step 70: determining the matching result of the SAR image and the visible light image according to the data set to be matched, the feature extraction network and the updated feature mapping layer.

2. According to claim 1, a heterogeneous image template matching method based on feature adversarial network is characterized in that: The feature extraction network includes a first feature extraction subnetwork and a second feature extraction subnetwork with the same structure, and the network weights of the first feature extraction subnetwork and the network weights of the second feature extraction subnetwork are not shared; The first feature extraction subnetwork is used to extract features from the SAR image and output a SAR image feature map sequence; The second feature extraction subnetwork is used to extract features from the visible light image and output a visible light image feature map sequence; The feature mapping layer includes a first mapping subnetwork and a second mapping subnetwork with the same structure; The first mapping subnetwork is used to map the SAR image feature map sequence and output a SAR mapping feature map sequence; The second mapping subnetwork is used to map the visible light image feature map sequence and output a visible light mapping feature map sequence.

3. The heterogeneous image template matching method based on feature adversarial network according to claim 2 is characterized in that: The step 30 comprises: Step 31: input the SAR image and the corresponding visible light image of the image pair of the training set into the first feature extraction subnetwork and the second feature extraction subnetwork respectively, and output SAR image feature map sequences respectively. And the visible light image feature map sequence Among them, s i represents the feature map of the i-th SAR image in the training set, o i represents the feature map of the i-th visible light image in the training set, and n represents the number of image pairs consisting of SAR images and visible light images; Step 32: input the SAR image feature map sequence and the visible light image feature map sequence into the first mapping subnetwork and the second mapping subnetwork respectively, and output a sequence of SAR mapping feature maps. and a sequence of visible light mapping feature maps where f S and f O denote the mapping function of the first mapping sub-network and the mapping function of the second mapping sub-network respectively, θ S represents the network parameters of the first mapping subnetwork, θ O The network parameters of the second mapping sub-network are represented.

4. The heterogeneous image template matching method based on feature adversarial network according to claim 3 is characterized in that: The step 40 comprises: Step 41: The sequence SP of the SAR mapping feature graph S and the image sequence SP of the visible light mapping feature O The modal discrimination network is input for classification, and the discrimination loss is calculated; the loss function of the discrimination loss is: Where D represents the discriminator of the modality discrimination network, θ D Represents the network parameters of the mode discrimination network; Step 42, in a feature mapper, mapping the sequence SP of the feature map according to the SAR S and the sequence SP of the visible light mapping feature map O Calculate SP S and SP O The L2 norm between: L2=||SP s -SP o ||2; Step 43: Calculate the entropy L of the total gray-level co-occurrence matrix in the feature mapper according to the sequence of the SAR mapping feature graphs and the sequence of the visible light mapping feature graphs. Ent : Among them, f Ent Represents the mapping function of all feature channel images of a feature map to the mean entropy of its corresponding gray-level co-occurrence matrix; Step 44, in the feature mapper, the matching loss is calculated according to the SAR image mapping feature map, the visible light mapping feature map and the matching label: the Euclidean distance between the SAR image template area corresponding to the SAR image mapping feature map and the visible light mapping feature map is calculated by sliding window as a similarity index, and the upper left corner point of the SAR image template corresponding to the minimum Euclidean distance is taken as the matching point g i ', calculate g i 'Image matching label g corresponding to the SAR image template i The loss is also the matching loss, and the loss function of the matching loss is: Among them, MSE is the mean square error loss function.

5. The heterogeneous image template matching method based on feature adversarial network according to claim 3 is characterized in that: The total loss of the mapping is: Among them, α and β are hyperparameters, L r represents the regular function, represents the parameters of the first layer of the first mapping subnetwork, N Layer Indicates the number of network layers of the mapping subnetwork. Represents the parameters of the lth layer of the second mapping subnetwork.

6. The heterogeneous image template matching method based on feature adversarial network according to claim 4, characterized in that: The step 60 comprises: Step 61, constructing an objective function according to the total mapping loss and the discrimination loss; wherein the expression of the objective function is: in, represents the updated network parameters of the first mapping subnetwork, represents the updated network parameters of the second mapping subnetwork, Represents the network parameters of the updated modal discrimination network; Step 62: According to the objective function, the Adam algorithm is used to update the parameters of the feature mapping layer and the modal discrimination network to obtain an updated feature mapping layer and an updated modal discrimination network.

7. The heterogeneous image template matching method based on feature adversarial network according to claim 4, characterized in that: The step 70 includes: inputting the SAR image and the visible light image in the to-be-matched data set into the feature extraction network and the updated feature mapping layer in sequence to obtain a sequence of SAR target mapping feature maps and a sequence of visible light target mapping feature maps, and then calculating the Euclidean distance between the SAR target mapping feature map and the visible light target mapping feature map by means of a sliding window, and obtaining the coordinates of the upper left corner pixel point of the SAR target mapping feature map corresponding to the minimum Euclidean distance in the visible light target mapping feature map.

Images