Infrared image domain self-adaption method and system based on map feature alignment

Abstract

The invention discloses an infrared image domain adaptive method and system based on map feature alignment, and mainly solves the problems of poor generalization ability and low classification precision in the prior art. According to the implementation scheme, the method comprises the following steps: selecting an antagonism domain adaptive model to extract source domain and target domain features; calculating the sum of supervised classification loss and neighborhood adversarial loss by using the extracted features as initial loss; calculating the difference between the features, and abstracting the difference into a graph structure; calculating the distance between the source domain graph and the target domain graph by using a difference measurement function, and adding the distance into an initial loss function to obtain intermediate loss; using a K nearest neighbor algorithm in the aligned target domain to obtain a pseudo label and confidence of each sample, adding the pseudo label and the confidence into an intermediate loss function to obtain final loss, updating domain adaptive model parameters through back propagation, and comparing the pseudo label with a real label to obtain the classification accuracy of the target domain. According to the method, the calculation amount is greatly reduced, the classification accuracy of the target domain is improved, and the method can be used for infrared target identification and classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision classification, and in particular relates to an infrared image domain adaptive method and system, which can be used for infrared target recognition and classification, and improve the model's ability to recognize targets in different environments. Background Art

[0002] In recent years, with the continuous progress of machine learning technology, it has become possible to use computers to perform target recognition and classification tasks. Many methods based on convolutional neural networks and visual converters have made significant progress in many computer vision tasks. However, these proposed methods are based on the assumption that the training data set and the test data set are independent and identically distributed, that is, there must be enough similarity between the two data sets, which means that when the environment changes, such as changes in lighting and background, it will cause a large difference in the data distribution of the training set and the test set, that is, domain shift, which will lead to a significant decrease in the accuracy of model prediction.

[0003] In order to solve the above problems, domain adaptation technology has been proposed to solve the situation where the distribution of training data and test data is different. The main idea of ​​domain adaptation is to migrate the source domain information containing rich annotation information to the target domain, so as to assist the model in learning on the target domain. According to the annotation of the data in the target domain, domain adaptation can be divided into three categories: supervised domain adaptation, whose target domain contains only a small amount of labeled data; semi-supervised domain adaptation, whose target domain contains a small amount of labeled data and a large amount of unlabeled data; unsupervised domain adaptation, whose target domain contains only a large amount of unlabeled data. Compared with supervised domain adaptation and semi-supervised domain adaptation, unsupervised domain adaptation has become the focus of research because it does not require manual annotation at all.

[0004] Existing unsupervised domain adaptation methods include: difference-based methods, adversarial adaptation methods, and self-training methods. The difference-based method minimizes the distribution difference between the source domain and the target domain through difference metrics; the adversarial adaptation method learns domain-invariant features through adversarial training between feature extractors and domain classifiers; and the self-training method improves the model by generating pseudo labels for target domain data.

[0005] The patent document with application number CN202110474134.7 discloses "a cross-domain infrared target detection method", whose main steps are: (1) obtaining labeled source domain data and target domain data with a small amount of labels and a large amount of unlabeled data; (2) using the labeled source domain data to train the Mask R-CNN-1 network; (3) using the source domain data and the target domain data to train a new Mask R-CNN-2 network; (4) inputting the target to be detected into the trained Mask R-CNN-2 network, and finally realizing the target detection function. This method uses domain adaptation technology to improve the network training effect and target detection accuracy when the target domain data labels are insufficient; it supplements the shortcomings of performing domain adaptation tasks only at the feature level, uses the Mask R-CNN network, and goes a step further on the basis of target detection to achieve pixel-level target detection. However, this method belongs to semi-supervised domain adaptation, and still requires some labeled target domain data. It may not improve the network training effect in a dataset without any labels.

[0006] The patent document with application number CN201810400557.2 discloses a "robust infrared target recognition method integrating multi-feature dimensionality reduction and transfer learning". It firstly addresses the problem that the traditional infrared target feature extraction method does not cover all information when extracting a single feature, and proposes to extract heterogeneous features of different types of targets, including shape features and brightness features of target images, so as to fully explore the characteristics of infrared targets. Secondly, it proposes to use the principal component analysis method to reduce the dimensionality of the fused heterogeneous features. Finally, an effective infrared target classifier based on transfer learning is designed to maximize the generalization performance and target recognition accuracy. Although this method improves the performance of infrared target recognition under complex backgrounds compared with traditional infrared target recognition methods, it is not suitable for pure infrared band application scenarios because it relies on visible light image datasets to assist in model training. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the prior art and propose an infrared image domain adaptive method and system based on atlas feature alignment to solve the problems of poor adaptability of existing infrared target classification methods and low accuracy in target recognition and classification under different environments.

[0008] The idea of the present invention is to extract the features of the source domain and the target domain by using a residual neural network. Taking each eigenvalue as a vertex, a similarity metric function is used to calculate the difference value between every two vertices, and this difference value is used as the value of the weighted edge between two points, constructing a graph composed of the extracted eigenvalues to measure the distribution difference between the source domain and the target domain. By using the Laplacian matrix to calculate the distance between the source domain graph and the target domain graph, it is added as part of the loss to the overall loss to improve the model. Finally, the K-nearest neighbor algorithm is used to obtain the prediction results of the target domain data, and the confidence of the prediction results is added to the overall loss to improve the model, obtaining a classification result with high accuracy.

[0009] According to the above idea, the technical solution of the present invention includes the following:

[0010] 1. An infrared image domain adaptation method based on graph spectrum feature alignment, characterized by comprising:

[0011] 1) Select an adversarial domain adaptation model and use its residual neural network to extract the eigenvalue of the infrared image source domain dataset and the eigenvalue of the infrared image target domain dataset respectively;

[0012] 2) Taking each extracted eigenvalue as a vertex, a similarity metric function is used to calculate the difference value σ between every two vertices, and using the difference value σ as the value of the weighted edge between two vertices, respectively construct a source domain feature distribution graph and a target domain feature distribution graph, and store the topological relationship of the graph in the form of an adjacency matrix;

[0013] 3) Calculate the graph spectrum distance δ between the source domain feature distribution graph and the target domain feature distribution graph, and use the graph spectrum distance value δ as the graph spectrum alignment loss to reduce the distance between the graphs, and add it to the loss function of the adversarial domain adaptation model to optimize the global consistency of the source domain and target domain feature distributions, and complete the coarse-grained domain alignment of the source domain and the target domain;

[0014] 4) Use the K-nearest neighbor algorithm to calculate the confidence of each data in the target domain data after coarse-grained domain alignment, use this confidence value as the neighborhood-aware propagation loss, and add it to the intermediate loss function to obtain the final loss function

[0015] to measure the difference between the prediction result and the actual result, and update the model parameters through backpropagation to realize the migration of the infrared image source domain dataset and the target domain dataset to the same distribution, and improve the target recognition and classification accuracy.

[0016] According to the source domain feature distribution diagram G s 's adjacency matrix A s and the target domain feature distribution diagram G t 's adjacency matrix A t Calculate the Laplacian matrix L of A s and the Laplacian matrix L of A s and A t 's Laplacian matrix L t :

[0017] Convert the above two Laplacian matrices L s and L t into symmetric normalized Laplacian matrices and

[0018] According to the normalized Laplacian matrices and Calculate the eigenvalues λ of the two Laplacian matrices s and λ t , and obtain the eigenvalue sets Λ of the two Laplacian matrices s and Λ t

[0019] According to the eigenvectors Λ of the two Laplacian matrices s and Λ t , define the spectral distance δ between the source domain feature distribution diagram G s and the target domain feature distribution diagram G t , and add it to the loss function of the adversarial domain adaptation model to obtain the intermediate loss function

[0020] Furthermore, the use of the K-nearest neighbor algorithm to calculate the confidence of each data in the target domain after coarse-grained domain alignment is for each target domain sample X i , using the feature extractor F(·) and the class discriminator C(·) to obtain the prediction probability P of each sample X i and add the prediction probabilities P of the first K closest other target domain samples X i,c to obtain the confidence Q of the data sample X j with the prediction result of class c j,c , and then normalize it to obtain the normalized confidence i , i,c , and then normalize it to obtain the normalized confidence

[0021] Furthermore, the addition of the confidence value as the neighborhood-aware propagation loss to the intermediate loss function is to represent the prediction result of the target domain data sample as: According to Define the neighborhood propagation loss Add it to the intermediate loss function to obtain the final loss function

[0022] 2. An infrared image domain adaptation system based on spectral feature alignment, characterized in that it includes:

[0023] A feature extraction module, which is used to extract the feature values of the source domain and the target domain, and calculate the supervised classification loss and the domain adversarial loss;

[0024] A spectral map generation module, based on the extracted feature values of the source domain and the target domain, uses a similarity metric function to calculate the difference values between the feature values, takes the feature values as vertices, and the difference values between the feature values as weighted edges, abstracts the relationship between the feature values into a graph structure, and stores the topological information of the graph with an adjacency matrix;

[0025] A spectral map alignment module, which is used to calculate the distance between two graphs according to the generated source domain feature distribution map and target domain feature distribution map, and add this distance as the spectral map alignment loss to the total loss;

[0026] A neighborhood awareness module, which is used to predict the category of each target in the target domain using the K-nearest neighbor algorithm after the rough alignment of the source domain feature distribution and the target domain feature distribution, and use the confidence of the prediction result as the neighborhood propagation loss to be added to the total loss;

[0027] A category prediction module, which uses a loss function to measure the difference between the prediction result and the actual result, and updates the model parameters through backpropagation. In the target domain after the model parameters are updated, it uses a class discriminator to predict the probability that a target domain sample belongs to a certain class in the target domain, and uses the K-nearest neighbor algorithm to obtain the class with the highest probability, and outputs this class as the classification result of the target domain sample.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] First, since the present invention adopts an unsupervised domain adaptation method and does not require a labeled target domain, the manual annotation work is reduced;

[0030] Second, since the present invention focuses on the features of infrared images and does not require visible light data for auxiliary training;

[0031] Third, since the present invention uses an overall spectral map alignment strategy, compared with the point-to-point alignment strategy, the amount of calculation is greatly reduced;

[0032] Fourthly, since the K-nearest neighbor algorithm is adopted in the present invention to obtain the classification prediction result of the target domain, the obtained classification result has higher accuracy compared with directly using a class discriminator for classification. Description of the Drawings

[0033] Figure 1 is the implementation flowchart of the infrared image domain adaptation method based on atlas feature alignment in Example 1 of the present invention;

[0034] Figure 2 is Figure 1 the schematic diagram of feature map construction in;

[0035] Figure 3 is Figure 1 the schematic diagram of atlas alignment process in;

[0036] Figure 4 is Figure 1 the schematic diagram of neighborhood-aware propagation in;

[0037] Figure 5 is the structural block diagram of the infrared image domain adaptation system based on atlas feature alignment in Example 2 of the present invention. Detailed Embodiments

[0038] The technical solutions of the present invention will be further described in detail below with reference to the drawings and embodiments:

[0039] In order to enable those skilled in the art of the present technology to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the invention examples. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0040] Example 1, an infrared image domain adaptation method based on atlas feature alignment.

[0041] Referring to Figure 1 , the implementation steps of this example are as follows:

[0042] Step 1, feature extraction

[0043] The residual neural network can solve the problem of gradient disappearance or gradient explosion that occurs in deep neural networks when the number of layers is large. Therefore, it is very suitable for the infrared image classification task that requires deep neural networks for feature extraction.

[0044] The training set data with label annotations is called the source domain S, and the target domain data without label annotations is called the target domain T. The data samples in the source domain and the target domain are denoted as Xs and Xt;

[0045] Use a residual neural network as the feature extractor F(·), and extract the source domain feature Fs and the target domain feature Ft from the source domain sample Xs and the target domain sample Xt respectively:

[0046] Fs = F(Xs)

[0047] Ft = F(Xt)

[0048] In this example, a 50-layer residual neural network or a 101-layer residual neural network is used as the feature extractor F(·) for feature extraction. The feature extracted by the 50-layer residual neural network is called Fs 50 and Ft 50 , and the feature extracted by the 101-layer residual neural network is called Fs 101 and Ft 101 .

[0049] Step 2: Calculate the supervised classification loss and the domain adversarial loss of the adversarial domain adaptation to obtain the loss function of the adversarial domain adaptation model

[0050] The so-called adversarial domain adaptation is to make the domain classifier unable to distinguish whether the data comes from the source domain or the target domain through adversarial learning, so as to learn the common features of the source domain and the target domain and obtain the domain adversarial loss The supervised classification loss is obtained by training the source domain data with labels Its specific implementation includes the following:

[0051] 2.1) Use the feature extractor F(·) and the class classifier C(·) to generate the supervised classification loss

[0052] The class classifier C(·) adopts a fully connected layer neural network and a Softmax activation function. The input is the output of the feature extractor F(·), and the output is the class probability.

[0053] Use the cross-entropy loss function to calculate the difference between the predicted probability of the source domain data and the true label of the source domain ; for all source domain data, use the cross-entropy loss function to obtain the difference between the predicted result and the true label, and take its average value as the supervised classification loss of the adversarial domain adaptation

[0054]

[0055] Among them, E represents the expectation of the source domain data distribution , that is, the average of the losses of all source domain samples;

[0056] 2.2) Generate the domain adversarial loss using the feature extractor F(·) and the domain classifier D(·)

[0057] The domain classifier D(·) is a binary classifier. The input is the output of the feature extractor F(·), and the output is a probability value. The closer the output is to 0, the more likely the data is from the source domain, and the closer the output is to 1, the more likely the data is from the target domain;

[0058] Let represent the probability that the domain classifier D(·) recognizes that the source domain features come from the source domain. Apply to each source domain data and take the average value as the first part of the domain adversarial loss;

[0059] Let represent the probability that the domain classifier D(·) recognizes that the target domain features come from the target domain. Apply to each target domain data and take the average value as the second part of the domain adversarial loss;

[0060] Add the first part and the second part to obtain the domain adversarial loss

[0061]

[0062] where is the data of the target domain, and represent the source domain data and the target domain data feature distributions respectively;

[0063] 2.3) Obtain the initial loss function according to the supervised classification loss and the domain adversarial loss

[0064]

[0065] Step three, construct the feature map.

[0066] Using the graph data structure can effectively represent the rich spatial structure contained in the image data. It can not only model the relationships between different samples within the domain and represent the underlying feature distribution, but also facilitate the computer to process the graph-structured data, which is beneficial to simplify the calculation.

[0067] Refer to Figure 2 This step is implemented as follows:

[0068] 3.1) Using the extracted source domain features Fs, the i-th feature vector and the j-th feature vector As a pair of vertices, calculate the difference between the two vertices using a similarity metric function As a weighted edge:

[0069] where ∥·∥ represents the norm of the feature vector, i, j ∈ Ns, and Ns is the number of source domain features extracted;

[0070] Take all the source domain features as vertices, and the differences between the vertices as weighted edges to form the source domain graph Gs.

[0071] 3.2) Using the extracted target domain features Ft, for the i-th feature vector among them and the j-th feature vector As a pair of vertices, calculate the difference between the two vertices using a similarity metric function As a weighted edge:

[0072] where i, j ∈ Nt, Nt represents the number of target domain features extracted, and Nt is the same as Ns;

[0073] Take all the target domain features as vertices, and the differences between the vertices as weighted edges to construct the target domain graph Gt.

[0074] 3.3) Represent the adjacency matrices of the source domain graph Gs and the target domain graph Gt as As and At respectively. The adjacency matrix of the graph contains all the topological information of the graph. Among them, the element in the i-th row and j-th column of the source domain graph adjacency matrix A s in the source domain is used to store the difference value between two features in the source domain t The element in the i-th row and j-th column of the target domain graph adjacency matrix A is used to store the difference value between two features Using the source domain graph and the target domain graph, not only can the intra-domain relationships be obtained, but also the inter-domain alignment can be transformed into a graph alignment problem.

[0075] Step Four, graph alignment.

[0076] Since the target domain feature distribution graphs Gs and Gt constructed by the above-mentioned source domain features Fs and target domain features Ft are quite different, which will affect the accuracy of classification in the target domain. Therefore, it is necessary to perform inter-domain alignment on them.

[0077] Existing inter-domain alignment requires point-to-point graph matching, which involves solving the combined optimization problems of node matching and edge matching respectively, and the calculation process is complex. Therefore, this example adopts the overall graph alignment method to avoid the complex steps of point-to-point graph matching, thereby simplifying the method for measuring the differences between graphs.

[0078] Refer to Figure 3 , the implementation of this step includes the following:

[0079] 4.1) According to the adjacency matrix A s of the source domain feature distribution graph G s and the adjacency matrix A t of the target domain feature distribution graph G t respectively calculate the Laplacian matrix L s of A s and the Laplacian matrix L t of A t :

[0080] L s = D s - A s

[0081] L t = D t - A t

[0082] where D s is the degree matrix of the adjacency matrix A s of the source domain feature distribution graph, and D t is the degree matrix of the adjacency matrix A t of the target domain feature distribution graph;

[0083] 4.2) Transform the above two Laplacian matrices L s and L t into symmetric normalized Laplacian matrices and which are expressed as follows:

[0084]

[0085] where I is the identity matrix;

[0086] 4.3) According to the normalized Laplacian matrix and the identity matrix I, calculate the eigenvalues λ of the normalized Laplacian matrix s and of the eigenvalues λ t respectively through the following formula:

[0087]

[0088] The normalized Laplacian matrix with its n eigenvalues constitute a set Λ s , and the n eigenvalues λ of the normalized Laplacian matrix t constitute a set Λ t , which are respectively denoted as:

[0089]

[0090] where represents the a-th eigenvalue of the normalized Laplacian matrix , represents the a-th eigenvalue of the normalized Laplacian matrix , and n represents the dimension of the normalized Laplacian matrix and ;

[0091] 4.4) According to the eigenvectors Λ s and Λ t of the two Laplacian matrices described above, define the graph distance δ between the source domain feature distribution graph G s and the target domain feature distribution graph G t as:

[0092] δ = ∥Λ s - Λ t ∥ p

[0093] where ∥·∥ p represents the p-norm, which is used to measure the difference between two sets of eigenvalues. p = 1 is the Manhattan distance, p = 2 is the Euclidean distance, and p = ∞ is the maximum difference;

[0094] 4.5) Take the graph distance value δ as the graph alignment loss, that is Then add this graph alignment loss to the initial loss function to obtain the intermediate loss function for optimizing the global consistency of the source domain and target domain feature distributions, which is expressed as follows:

[0095]

[0096] where is the supervised classification loss, is the domain adversarial loss.

[0097] Step Five, neighborhood-aware propagation, to obtain the final loss function of the adversarial domain adaptation model

[0098] After the above spectral alignment step, the rich topological information in the source domain is roughly transferred to the target domain. To perform fine-grained intra-domain alignment, the K-nearest neighbor classification algorithm is used to generate pseudo-labels for the target domain graph, promote the transmission of information in the target domain graph, and obtain the final loss function of the adversarial domain adaptation model.

[0099] Refer to Figure 4 , the implementation of this step is as follows:

[0100] 5.1) For each target domain sample X i , use the feature extractor F(·) and the class discriminator C(·) to obtain the prediction probability P i,c = C(F(X i )) of each sample Xi, where c ∈ C t , and C t is the number of classes in the target domain;

[0101] 5.2) Add the prediction probabilities P i of the top K samples X j closest to other target domain samples X j,c to obtain the confidence Q i that the data sample X i,c is of class c:

[0102]

[0103] where is the neighborhood of the data sample X i ;

[0104] 5.3) Normalize the confidence Q i,c to obtain the normalized confidence

[0105]

[0106] 5.4) Represent the pseudo-label of the data sample Xi as Use the normalized confidence as the confidence value of each pseudo-label to obtain the neighborhood propagation loss

[0107]

[0108] where is the confidence of the pseudo-label , and is the prediction of the sample X i as class The probability; α is a dynamic coefficient, with a value range of [0.5, 2.0]. It is smaller in the initial stage of iteration to suppress early noise, linearly increases gradually with iteration, and reaches the maximum value in the last iteration to enhance the supervision signal;

[0109] 5.5) Add the neighborhood propagation loss to the intermediate loss function to obtain the final loss function of the adversarial domain adaptation model

[0110]

[0111] Step six, train the adversarial domain adaptation model to obtain the classification accuracy of the target domain.

[0112] 6.1) Take the source domain data Xs and the target domain data Xt as inputs and input them into the adversarial domain adaptation model;

[0113] 6.2) Set the number of training times to 1400, the batch sizes of the source domain and the target domain to 32, the K value of the K-nearest neighbor algorithm to 5, and the neighborhood-aware propagation loss coefficient to 0.5;

[0114] 6.3) Calculate the gradient of the loss function with respect to the parameters of the domain adaptation model, transmit the gradient through the backpropagation algorithm, use the optimizer to update the parameters of the domain adaptation model according to the gradient direction, gradually reduce the loss value, and calculate the pseudo-label of the target domain sample Xi in the current round Compare it with the true label of Xi to obtain the classification accuracy of the target domain in this round;

[0115] 6.4) Repeat step 6.3) until the loss function converges or reaches the maximum number of iterations, end the training, and output the classification accuracy of the target domain in each round;

[0116] Example 2, an infrared image domain adaptation system based on spectral feature alignment.

[0117] Refer to Figure 5 , this example includes a feature extraction module 1, a spectral map generation module 2, a spectral map alignment module 3, a neighborhood awareness module 4, and a class prediction module 5. Its working principle is as follows:

[0118] The feature extraction module 1 is used to extract the feature values of the source domain and the target domain, calculate the supervised classification loss and the domain adversarial loss, output the feature values of the source domain and the target domain to the spectral map construction module 2, and output the supervised classification loss and the domain adversarial loss to the spectral map alignment module 3;

[0119] The graph construction module 2 calculates the difference values between the feature values based on the source domain and target domain feature values extracted by the feature extraction module 1 using a similarity metric function. Taking the feature values as vertices and the difference values between the feature values as weighted edges, it abstracts the relationship between the feature values into a graph structure and stores the topological information of the graph in an adjacency matrix, and then outputs the adjacency matrix storing the graph topological information to the graph alignment module 3;

[0120] The graph alignment module 3 calculates the distance between the two graphs according to the source domain feature distribution map and target domain feature distribution map generated by the graph construction module 2, and takes this distance as the graph alignment loss to be added to the supervised classification loss and domain adversarial loss output by the feature extraction module 1. Reducing this loss can narrow the distance between the source domain graph and the target domain graph to achieve graph alignment, and then outputs the aligned target domain and loss function to the neighborhood perception module 4;

[0121] The neighborhood perception module 4 is used to predict the category of each target in the target domain output by module 3 using the K-nearest neighbor algorithm in the target domain, use the confidence of the prediction result as the neighborhood propagation loss to be added to the loss of the graph alignment module 3, and output this loss function to the category prediction module 5;

[0122] The category prediction module 5 uses the loss function obtained through the neighborhood perception module 4 to measure the difference between the prediction result and the actual result, updates the model parameters through backpropagation. In the target domain after the model parameters are updated, it uses the K-nearest neighbor algorithm to obtain the pseudo-labels of the samples, and compares them with the true labels of the samples to obtain the target domain classification accuracy and outputs it.

[0123] The effects of the present invention can be further illustrated by the following simulation experiments:

[0124] I. Simulation conditions

[0125] Hardware conditions:

[0126] CPU: Intel(R) Xeon(R) Platinum 8358P CPU @ 2.60GHz, 32 cores, 128

[0127] processors

[0128] GPU: NVIDIA GeForce RTX 3060

[0129] Feature extractor: Use a 50-layer residual neural network as the feature extractor;

[0130] Parameter settings: In the K-nearest neighbor algorithm, the value of K is 5, and the initial coefficient in the loss function is 0.5, and the size of each batch is set to 32;

[0131] II. Simulation Content and Results

[0132] Using the infrared simulation dataset as the source domain, 960 images with 160 images of each type of M1A1, T72b, 2S3, M109, BMP3, and BTR82 are selected as training samples. Using the mid-wave infrared dataset MWIR as the target domain, 600 images with 100 images of each type of M1A1, T72b, 2S3, M109, BMP3, and BTR82 are selected as target samples. Different loss functions are used to update the model parameters, classify the samples in the target domain, and compare them with the true categories to obtain the classification accuracy. The results are shown in Table 1.

[0133] Table 1 Classification Accuracy (%) of the Present Invention on the Infrared Image Dataset

[0134]

[0135] In the table is the supervised classification loss, is the initial loss of the adversarial domain adaptation model, is the final loss of the present invention.

[0136] It can be seen from Table 1 that the method of only using to improve the model parameters has a final accuracy of 31.9% because it does not perform domain adaptation between the source domain and the target domain, and the model trained in the source domain is difficult to distinguish the features of the target domain data. The method using the loss of the adversarial domain adaptation method to improve the model parameters has an accuracy increased to 47.1%, indicating that this method can effectively learn the common features of the source domain and the target domain. The method using the loss function proposed by the present method to improve the model parameters has an accuracy of 64.0%.

[0137] The simulation results show that: based on the adversarial domain adaptation method, the present invention further realizes the alignment of the source domain feature distribution and the target domain feature distribution, and can greatly improve the classification accuracy in the target domain.

[0138] It should be noted that the step numbers in the specification and claims of the present invention are only for a clear description of the embodiments of the present invention for easy understanding, and their sequence numbers are not limited.

Claims

1. A method for infrared image domain adaptation based on atlas feature alignment, characterized in that: include: 1) Select the adversarial domain adaptation model, use the residual neural network to extract the eigenvalues ​​of the infrared image source domain dataset and the eigenvalues ​​of the infrared image target domain dataset, and calculate the supervised classification loss and domain adversarial loss; 2) Taking each extracted eigenvalue as a vertex, using a similarity measurement function to calculate the difference value σ between every two vertices, and taking the difference value σ as the value of the weighted edge between the two vertices, constructing a source domain feature distribution graph and a target domain feature distribution graph respectively, and storing the topological relationship of the graph in the form of an adjacency matrix; 3) Calculate the spectral distance δ between the source domain feature distribution map and the target domain feature distribution map, use the spectral distance value δ as the spectral alignment loss to reduce the distance between the maps, and add it to the loss function of the adversarial domain adaptation model In the example above, we get the intermediate loss function Optimize the global consistency of feature distribution between the source domain and the target domain, and complete the coarse-grained domain alignment between the source domain and the target domain; 4) Use the K nearest neighbor algorithm to calculate the confidence of each data in the target domain data after the coarse-grained domain alignment, use the confidence value as the neighborhood-aware propagation loss, and add it to the intermediate loss function The final loss function is obtained It is used to measure the difference between the predicted results and the actual results, and to update the model parameters through back propagation, so as to migrate the infrared image source domain dataset and the target domain dataset to the same distribution, thereby improving the accuracy of target recognition and classification.

2. The method according to claim 1, characterized in that The adversarial domain adaptation model selected in step 1) includes: Supervised classification loss: Field adversarial loss: in, and are the data of the source domain and the target domain respectively. is the data label of the source domain, F(·) is the feature extractor, C(·) is the class discriminator, D(·) is the domain discriminator, Represents the source domain data D S expectations, is the cross entropy loss function, and Represent the source domain data D S and target domain data D T The characteristic distribution of Denotes the domain classifier D(·) to identify the source domain features The probability of coming from the "source domain", Denotes the domain classifier D(·) that identifies the target domain features Probability of being from the "target domain". Loss Function for Adversarial Domain Adaptation Model It is expressed as:

3. The method according to claim 1, characterized in that Step 1) The residual neural network in the adversarial domain adaptation model is used to extract the eigenvalues ​​of the infrared image source domain dataset and the eigenvalues ​​of the infrared image target domain dataset respectively. A 50-layer residual neural network or a 101-layer residual neural network that can balance the amount of computation and accuracy is selected as a feature extractor to extract the source domain and target domain features of the infrared image.

4. The method according to claim 1, characterized in that In step 2), the extracted source domain feature Fs is used to transform the i-th feature vector and the jth eigenvector As a pair of vertices, the difference between the two vertices is calculated using the similarity measurement function As a weighted edge: Where ||·|| represents the norm of the feature vector, i, j∈Ns, and Ns is the number of extracted source domain features; Using the extracted target domain features Ft, the i-th feature vector and the jth eigenvector As a pair of vertices, the difference between the two vertices is calculated using the similarity measurement function As a weighted edge: Among them, i, j∈Nt, Nt represents the number of extracted target domain features, and Nt is the same as Ns.

5. The method according to claim 1, characterized in that In step 2), the source domain feature distribution graph and the target domain feature distribution graph are constructed, and the edge information of the graph is stored in an adjacency matrix. The implementation includes the following: 2a) All source domain feature values ​​extracted in step 1) and As vertices, the difference between two vertices As a vertex and The values ​​of the weighted edges between them constitute the source domain feature distribution graph G s ; To extract all the target domain features and As vertices, the difference between two vertices As a vertex and The values ​​of the weighted edges between them constitute the target domain feature distribution graph G t ; 2b) Using the adjacency matrix A s and A t Store the source domain feature distribution map G separately s And the target domain feature distribution map G t The side information of A s The i-th row and j-th column in Store two vertices and The difference between In A t The i-th row and j-th column in Store two vertices and The difference between 6. The method according to any one of claims 1 to 5, characterized in that: In step 3), the spectral distance δ between the source domain feature distribution map and the target domain feature distribution map is calculated, and its implementation includes the following: 3a) According to the source domain feature distribution map G s The adjacency matrix A s And the target domain feature distribution map G t The adjacency matrix A t Calculate A separately s The Laplace matrix L s and A t The Laplace matrix L t : L s =D s -A s L t =D t -A t Where D s is the adjacency matrix A of the source domain feature distribution graph s The degree matrix, D t is the adjacency matrix A of the target domain feature distribution graph t The degree matrix of 3b) Substitute the above two Laplace matrices L s and L t Transformed into a symmetric normalized Laplace matrix and It is expressed as follows: Where I is the identity matrix; 3c) According to the normalized Laplacian matrix and Calculate the eigenvalues ​​of the two Laplacian matrices λ respectively s and λ t , the formula is as follows: Where I is the identity matrix; The set of eigenvalues ​​of two Laplacian matrices Λ s and Λ t Respectively expressed as: Where n represents the dimension of the Laplacian matrix, and the eigenvalue 3d) According to the eigenvectors Λ of the two Laplace matrices s and Λ t , define the source domain feature distribution map G s And the target domain feature distribution map G t The spectral distance δ between them is: d=‖Λ s -L t ‖ p where ||·|| p It represents the p-norm, which is used to measure the size of the difference between two sets of eigenvalues, p = 1 is the Manhattan distance, p = 2 is the Euclidean distance, and p = ∞ is the maximum difference.

7. The method according to claim 1, characterized in that In step 3), the graph distance value δ is added as the graph alignment loss to the loss function of the adversarial domain adaptation model. Get the intermediate loss function It is used to optimize the global consistency of the feature distribution of the source domain and the target domain, which is expressed as follows: in, is the graph alignment loss, is the supervised classification loss, It is the field against loss.

8. The method according to claim 1, characterized in that: Step 4) Use the K nearest neighbor algorithm to calculate the confidence of each data in the target domain data after the coarse-grained domain alignment, and its implementation includes the following: 4a) For each target domain sample X i , use the feature extractor F(·) and class discriminator C(·) in step 1) to get each sample X i The predicted probability P i,c =C(F(X i )), where c∈C t , C t is the number of categories in the target domain; 4b) The top K closest samples from other target domains X j The predicted probability P j,c Add together to get data sample X i The prediction result is the confidence Q of category c i,c : in is the data sample X i Neighborhood of; 4c) Confidence Q i,c After normalization, the normalized confidence is obtained 9. The method according to claim 1, characterized in that: Step 4) Add the confidence value as the neighborhood-aware propagation loss to the intermediate loss function The implementation includes the following: The prediction result of the target domain data sample Xi It is expressed as: in For data sample X i The prediction result is the normalized confidence of category c; Defining Neighborhood Propagation Loss for: Among them, N t Indicates the total number of target domain data, is the data sample X i The prediction result is the category The normalized confidence of Predict the category for sample Xi The probability of , α is the dynamic coefficient; Propagate the loss to the neighborhood Add to intermediate loss function The final loss function is obtained 10. An infrared image domain adaptive system based on atlas feature alignment, characterized in that: include: Feature extraction module (1), used to extract feature values ​​of source domain and target domain, and calculate supervised classification loss and domain adversarial loss; Graph construction module (2): Based on the extracted source and target domain eigenvalues, the difference between the eigenvalues ​​is calculated using a similarity measurement function. The eigenvalues ​​are regarded as vertices and the difference between the eigenvalues ​​are regarded as weighted edges. The relationship between the eigenvalues ​​is abstracted into a graph structure, and the topological information of the graph is stored using an adjacency matrix. A graph alignment module (3) is used to calculate the distance between the two graphs based on the generated source domain feature distribution graph and the target domain feature distribution graph, and add the distance as the graph alignment loss to the total loss; Neighborhood perception module (4), used to predict the category of each target in the target domain using the K nearest neighbor algorithm after the feature distribution of the source domain and the feature distribution of the target domain are coarsely aligned, and the confidence of the prediction result is added to the total loss as the neighborhood propagation loss; The category prediction module (5) uses a loss function to measure the difference between the predicted result and the actual result, and updates the model parameters through back propagation. In the target domain after the model parameters are updated, the class discriminator is used to predict the probability that the target domain sample is a certain category in the target domain, and the K nearest neighbor algorithm is used to obtain the category with the highest probability, which is output as the classification result of the target domain sample.

Images