Hyperspectral single-source field generalization method based on training in test and storage medium

Abstract

The invention discloses a hyperspectral single-source field generalization method based on training during testing and a storage medium. The method comprises the following steps: randomizing an original hyperspectral image data field to construct a plurality of virtual source domains; and aligning the Hessian matrix of the virtual source domain to learn domain invariant knowledge, thereby determining a loss function and training a meta-source model. When the element target adapts, the maximum posterior probability value of the observation dynamic inference model of the virtual target domain is obtained through variational posterior reasoning according to a virtual target domain sample and a neighbor false label. In the element target test, on the basis of the maximum posterior probability value constraint, a virtual target domain sample and real label log likelihood are maximized to optimize an element source model, element target loss is deduced to determine an updating rule, and evaluation and optimization guidance are performed in a virtual target domain. In the element target test stage, neighbor pseudo labels are distributed for test target domain samples, and actual target domain sample prediction categories are obtained. According to the method, the cross-domain generalization ability of the model is improved under the condition that the target domain data source domain is inaccessible during training.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and in particular, relates to a hyperspectral single-source domain generalization method and storage medium based on test-time training. Background Art

[0002] In recent years, deep learning-based hyperspectral image (HSI) classification methods have demonstrated excellent performance in specific tasks. However, the success of these methods usually relies on the ideal assumption that the training data (source domain) and the test data (target domain) are identically distributed. This assumption is difficult to hold in practical applications. Affected by the acquisition time, geographical location, sensor differences, and environmental factors, there is usually a significant distribution deviation between different HSI data. Especially when the distribution difference between the source domain and the target domain is large, the performance of the model trained based on the source domain in the target domain is usually unsatisfactory. In many practical scenarios, the model needs to handle classification tasks in areas that were not observed during training.

[0003] Therefore, how to improve the cross-domain generalization ability of the model when the target domain data source is inaccessible during training has become a key issue to be solved in the field of HSI classification. Summary of the invention

[0004] The technical purpose of this application is to solve the problem of how to improve the cross-domain generalization ability of the model under the condition that the target domain data source domain is inaccessible during training.

[0005] To achieve the above purpose, the technical solution adopted in this application is:

[0006] In a first aspect, an embodiment of the present application provides a hyperspectral single-source domain generalization method based on test-time training, comprising the following steps:

[0007] Step 1: perform domain randomization on the original hyperspectral image data and construct multiple virtual source domains;

[0008] Step 2, learning domain invariant knowledge by aligning the Hessian matrices of each of the virtual source domains, determining a domain invariant loss function based on the domain invariant knowledge; and training a meta-source model based on the domain invariant loss function;

[0009] Step 3, performing a meta-target adaptation process: the meta-source model determines the observation dynamics of the virtual target domain through variational posterior inference based on the virtual target domain samples and the corresponding neighbor pseudo-labels, and then infers the maximum posterior probability value of the meta-source model in the virtual target domain;

[0010] Step 4, perform a meta-target testing process: based on the constraint of the maximum a posteriori probability value, optimize the meta-source model by maximizing the log-likelihood between the virtual target domain samples and the true labels under the meta-source model parameter conditions; based on the derived meta-target loss, determine the meta-source model update rule in the meta-target testing phase, evaluate the meta-source model loss in the virtual target domain, and guide the optimization of the meta-source model.

[0011] Step 5, performing the meta-target testing phase: using the meta-source model to assign neighbor pseudo labels to the test target domain samples, and obtaining the predicted categories corresponding to the actual target domain samples based on the neighbor pseudo labels.

[0012] Furthermore, the original hyperspectral image data is domain randomized to construct multiple virtual source domains, including:

[0013] The original hyperspectral image data is spectrally and / or spatially intervened by utilizing a combination of spectral perturbation, spectral sampling, spectral smoothing, pixel reorganization and spatial rotation to construct a plurality of virtual source domains.

[0014] Furthermore, one of the terms in the domain invariant loss function is the sth i Virtual source domain and s j Virtual source domain The cross-domain knowledge transfer measure between them;

[0015] The step 2 specifically includes: optimizing the cross-domain knowledge transfer measure to obtain an upper bound of the cross-domain knowledge transfer measure;

[0016] Based on the upper bound, the upper bound problem with a quadratic form is converted into a simple norm product relationship using the spectral norm to obtain a simplified upper bound; based on the simplified upper bound, the domain invariant loss function is updated; and the meta-source model is trained using the updated domain invariant loss function;

[0017] The updated feature extractor of the meta-source model is used to obtain virtual source domain features.

[0018] Furthermore, the meta-source model includes a feature extractor and a classifier; and step 3 specifically includes:

[0019] Inputting a virtual target domain sample into the feature extractor to obtain virtual target domain features;

[0020] Fitting the classifier using the virtual source domain features and the real labels, and using the fitted classifier to assign neighbor pseudo labels to the virtual target domain samples;

[0021] Calculate the category prototype based on the nearest pseudo-label and the virtual target domain feature, and obtain the category prediction of the meta-source model in the virtual target domain sample by matching the category prototype;

[0022] The neighbor pseudo-label is used as a supervisory signal, and the meta-source model is trained during testing using a cross entropy loss to obtain a maximum posterior probability value of the posterior distribution of the meta-source model in the virtual target domain.

[0023] Furthermore, the expression of the maximum a posteriori probability value is as follows:

[0024]

[0025] in is the maximum a posteriori probability value, Φ s is the source model parameter, x t is a sample of the target domain, is the predicted label of the target domain, λ 4 is the learning rate for training during testing, is the K nearest neighbor set of the source domain, is the i-th target sample of the K-th category, represents the gradient, L CE (·) is the cross entropy loss, P t is the category prototype of the target domain.

[0026] In a second aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of a hyperspectral single-source domain generalization method based on test-time training as provided in any possible implementation scheme of the first aspect.

[0027] Compared with the prior art, the beneficial technical effects achieved by the hyperspectral single-source domain generalization method based on test-time training provided in the embodiments of the present application include: an innovative gradient regularization strategy based on the Hessian matrix is ​​proposed, aiming to mine potential domain-invariant knowledge from gradient information and improve the generalization ability of the model; secondly, a test-time training mechanism based on Bayesian posterior parameter reasoning is developed, so that the domain generalization model can dynamically adapt to changes in data distribution in the target domain, thereby achieving personalized adaptation to the target domain; then, the gradient regularization and test-time training mechanisms are integrated into a meta-learning framework consisting of three stages: meta-source training, meta-target adaptation, and meta-target testing. By simulating the generalization scenario at test time in the source domain training stage, it is ensured that the model has robust generalization capabilities when facing unseen target domains. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic block diagram of the method principle provided in the embodiment of the present application. DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] In order to solve the problem of how to improve the cross-domain generalization ability of the model under the condition that the target domain data source domain is inaccessible during training, the embodiment of the present application provides a method for hyperspectral single-source domain generalization combined with a Bayesian meta-Hessian network. First, it breaks through the limitation of traditional methods that only focus on learning domain invariance, and innovatively proposes a gradient regularization strategy based on the Hessian matrix, aiming to mine potential domain-invariant knowledge from gradient information and improve the generalization ability of the model; secondly, a test-time training mechanism based on Bayesian posterior parameter reasoning is developed, so that the domain generalization model can dynamically adapt to the data distribution changes in the target domain, and realize personalized adaptation to the target domain; then, the gradient regularization and test-time training mechanism are integrated into a meta-learning framework consisting of three stages: meta-source training, meta-target adaptation, and meta-target testing. By simulating the generalization situation during the test in the source domain training stage, it is ensured that the model has a robust generalization ability when facing an unseen target domain.

[0031] The present application is further described below in conjunction with the accompanying drawings.

[0032] like Figure 1 As shown, the embodiment provides a hyperspectral single-source domain generalization method based on test-time training, including:

[0033] Step 1, in order to simulate the domain generalization scenario under the condition of a single source domain, the original hyperspectral image (HSI) data is subjected to diversified spectral and spatial interventions through domain randomization to construct multiple virtual domains, including virtual source domains and virtual target domains, to simulate the distribution changes between different domains.

[0034] Based on the original source domain D through domain randomization s Generate a virtual source domain s i is the virtual source domain number, N s is the number of virtual source domains.

[0035] The domain randomization used in some embodiments includes a combination of spectral perturbation, spectral sampling, spectral smoothing, pixel reorganization, spatial rotation, and pixel relocation.

[0036] Spectral perturbation: This operation simulates the potential spectral changes caused by environmental noise in cross-domain scenarios by adding noise to the original source domain data.

[0037] Spectral sampling: This translation operation simulates the spectral changes caused by differences in spectral resolution of different devices in cross-domain scenarios by sampling the spectral curve of the original source domain data at fixed intervals.

[0038] Spectral smoothing: This operation simulates the spectral changes caused by differences in sensor sensitivity in cross-domain scenarios by filtering and smoothing the spectral curve of the original source domain data.

[0039] Pixel reorganization: This operation simulates the distribution changes of objects in different HSIs in cross-domain scenes by disrupting the spatial order of each pixel in the original source domain data image block.

[0040] Spatial rotation: This operation simulates the perspective change when different HSIs are collected in cross-domain scenarios by rotating the original source domain hyperspectral image patches.

[0041] The method provided in this embodiment simulates different data distributions by constructing multiple virtual source domains, so that the model can be exposed to diverse data features during the training process without relying on the direct participation of the target domain data source during training. Even if the target domain data source is inaccessible during training, the model can still learn cross-domain knowledge through the virtual source domain.

[0042] Step 2: Based on each virtual source domain, meta-source training is performed. The model learns domain-invariant knowledge by aligning the Hessian matrices of different virtual source domains.

[0043] During meta-source training, some embodiments design a domain-invariant loss as follows:

[0044]

[0045] Among them, L D is the domain invariant loss value, and are the samples of the virtual source domain and the real virtual source domain labels, f(·) is the feature extractor, and c(·) is the KNN classifier. L CE (·) is the cross entropy loss, Virtual source domain and The cross-domain knowledge transfer measure between s is the number of virtual source domains.

[0046] The cross-domain knowledge transfer measure between different domains is maximized by optimizing its upper bound. The upper bound of can be expressed as:

[0047]

[0048] Among them, ζ is the δ-minimal set, π is the classifier parameter, and π *are the optimal parameters of the meta-source model in two virtual source domains, Indicates the virtual source domain The Hessian matrix of Indicates the virtual source domain The Hessian matrix of o(ζ 2 ) represents an infinitesimal quantity.

[0049] Then, the spectral norm is used to transform the upper bound problem with quadratic form into a simple norm product relation to simplify the calculation of the upper bound.

[0050] The domain invariant loss can be expressed as:

[0051]

[0052] Next, the meta-source model is trained based on the domain-invariant loss:

[0053]

[0054] Among them, Φ s' is the source model parameter, Φ s is the source model parameter, λ 1 is the learning rate of the source training phase, represents the gradient, represents the source domain label, x s Represents source domain samples.

[0055] Finally, the virtual source domain features are obtained using the feature extractor of the updated meta-source model.

[0056] The goal of the embodiment is to train an excellent source model so that it can perform well in the target domain. By simulating the test training scenario of the target domain in the source domain, the source model is trained based on virtual source domain samples and corresponding virtual source domain pseudo labels as supervision signals. The parameters of the source model are defined as the source model parameters Φ s Since there is no supervision signal in the target domain, a near-optimal model is trained, called the meta-source model. The parameters of the meta-source model are defined as the meta-source model parameters Φ s' .

[0057] Step 3, perform meta-target adaptation process: The meta-source model is based on the virtual target domain samples and the corresponding virtual target domain neighbor pseudo-labels, and calculates the maximum posterior probability value of the target domain meta-source model parameters through variational posterior inference to achieve intelligent adaptation to the target domain.

[0058] First, the feature extractor of the meta-source model is used to obtain the virtual target domain features. Then, the KNN classifier is fitted with the virtual source domain features and the corresponding real virtual target domain labels. By searching the K nearest neighbors of the virtual target domain samples in the virtual source domain, the virtual target domain nearest neighbor pseudo labels are assigned to the virtual target domain samples. Next, the category prototype is calculated based on the nearest virtual target domain neighbor pseudo labels and the virtual target domain features, and the category prediction of the model in the virtual target domain is obtained through prototype matching. Finally, the nearest neighbor pseudo labels are used as supervision signals, and the meta-source model is trained during testing using the cross entropy loss to obtain the posterior distribution p(Φ t' |P t' ,x t' ,Φ s' ) The optimization process of this stage can be expressed as:

[0059]

[0060] Among them, λ 2 is the learning rate of the meta-objective adaptation process, x t' is a virtual target domain sample, P t' is the category prototype of the virtual target domain, is the gradient, represents the K nearest neighbor set of the source domain, q θ (·) Conditional probability distribution.

[0061] Step 4, perform meta-target testing process: To further simulate the test-time training of the target domain, some embodiments utilize the maximum a posteriori probability value learned in the meta-target adaptation task to optimize the meta-source model by maximizing the log-likelihood between the target domain data and the true label under the conditions of the meta-source model parameters.

[0062] As an example, at this stage the model parameters are supervised by maximizing the following log-likelihood: Meta-target prediction process under the condition:

[0063]

[0064] Among them, maximize the first term This is equivalent to minimizing the model parameters as Under the condition of , the cross entropy loss of the model in the virtual target domain. Minimizing the second term is equivalent to minimizing the KL divergence between the category prototype distribution obtained based on the pseudo label and the category prototype distribution obtained based on the real label.

[0065] Therefore, the meta-objective loss can be obtained as follows:

[0066]

[0067] The update rule of the meta-goal testing phase model can be expressed as:

[0068]

[0069] Among them, λ 3 is the learning rate of the meta-objective test phase. It should be noted that L T is based on Calculated, but the update is in Φ s' By evaluating the loss of the model on the virtual target domain, the optimization of the meta-source model is guided, thereby improving its adaptability to the target domain that has not been seen in the meta-target testing phase.

[0070] In the embodiment, the loss function is determined by learning domain-invariant knowledge and the meta-source model is trained, as well as the optimization during meta-target adaptation and testing, which are all based on the general knowledge learned by the model and a small amount of target domain related information (such as virtual target domain samples and pseudo labels). The entire process does not rely on the accessibility of the target domain data source during training, but through clever design, the model is gradually adjusted at different stages to adapt to the target domain, thereby improving the cross-domain generalization ability of the model when the target domain data source is inaccessible.

[0071] In step 5, a nearest neighbor pseudo-labeling mechanism is proposed to map the target domain samples to the K nearest neighbor samples in the source domain, which reduces the generalization risk of the model in the target domain, improves the quality of the target domain pseudo-labels during the training process at test time, and helps the meta-source model to more accurately approximate the posterior distribution of the virtual target domain.

[0072] In the meta-target testing phase, the real labels of the target domain are inaccessible. The present invention calculates the category prototype based on the nearest pseudo-labels of the target domain samples, and then obtains the corresponding category prediction through the prototype metric. During the training at the test time, the cross entropy loss of the source model in the target domain is calculated using the nearest pseudo-labels to update the original model parameters to obtain The maximum a posteriori probability value

[0073]

[0074] Among them, λ 4 is the learning rate for testing training

[0075] Finally, based on Complete the prediction for the target domain.

[0076] In the embodiment of the present application, the meta-source training stage is used to simulate the domain-invariant knowledge learning process of the model in the source domain, and the gradient regularization constraint model has the same optimization path in different virtual source domains to minimize the upper bound of the cross-domain knowledge transfer measure between domains. The meta-target adaptation process is used to simulate the personalized adaptation process in the target domain. Based on Bayesian theory and variational reasoning, the posterior parameter distribution of the model in this domain is dynamically inferred according to the observation of the virtual target domain. The meta-target testing stage is used to simulate the testing process of the model in the target domain, and guides the optimization of the meta-source model by minimizing the prediction likelihood between the virtual target domain samples and labels under the conditions of the meta-source model parameters, thereby improving its adaptability to target domains not seen in the meta-target testing stage. Experimental results on multiple groups of HSI datasets show that the method proposed in this application has achieved state-of-the-art performance in generalization tasks in the hyperspectral field.

[0077] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the hyperspectral single-source domain generalization method based on test-time training as provided in the above embodiment are implemented.

[0078] The hyperspectral single-source field generalization method and storage medium based on test-time training provided by the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the concept of the present application and should not be understood as limiting the scope of protection of the present application.

Claims

1. A hyperspectral single-source domain generalization method based on test-time training, characterized in that: The steps include: Step 1: perform domain randomization on the original hyperspectral image data and construct multiple virtual source domains; Step 2, learning domain invariant knowledge by aligning the Hessian matrices of each of the virtual source domains, determining a domain invariant loss function based on the domain invariant knowledge; and training a meta-source model based on the domain invariant loss function; Step 3, performing a meta-target adaptation process: the meta-source model determines the observation dynamics of the virtual target domain through variational posterior inference based on the virtual target domain samples and the corresponding virtual target domain neighbor pseudo labels, and then infers the maximum posterior probability value of the meta-source model in the virtual target domain; Step 4, performing a meta-target testing process: based on the constraint of the maximum a posteriori probability value, optimizing the meta-source model by maximizing the log likelihood between the virtual target domain sample and the real label under the meta-source model parameter condition; determining the meta-source model update rule in the meta-target testing phase based on the derived meta-target loss, evaluating the meta-source model loss in the virtual target domain, and guiding the optimization of the meta-source model; Step 5, performing the meta-target testing phase: using the meta-source model to assign neighbor pseudo labels to the test target domain samples, and obtaining the predicted categories corresponding to the actual target domain samples based on the neighbor pseudo labels.

2. The hyperspectral single-source domain generalization method based on test-time training according to claim 1, characterized in that: The original hyperspectral image data is domain randomized to construct multiple virtual source domains, including: The original hyperspectral image data is spectrally and / or spatially intervened by utilizing a combination of spectral perturbation, spectral sampling, spectral smoothing, pixel reorganization and spatial rotation to construct a plurality of virtual source domains.

3. The hyperspectral single-source domain generalization method based on test-time training according to claim 1, characterized in that: The domain invariant loss function has an item in the sth i Virtual source domain and s j Virtual source domain The cross-domain knowledge transfer measure between them; The step 2 specifically includes: optimizing the cross-domain knowledge transfer measure to obtain an upper bound of the cross-domain knowledge transfer measure; Based on the upper bound, the upper bound problem with a quadratic form is transformed into a simple norm product relationship using the spectral norm to obtain a simplified upper bound; Update the domain invariant loss function based on the simplified upper bound; train the meta-source model using the updated domain invariant loss function; The updated feature extractor of the meta-source model is used to obtain virtual source domain features.

4. The hyperspectral single-source domain generalization method based on test-time training according to claim 1, characterized in that: The meta-source model includes a feature extractor and a classifier; the step 3 specifically includes: Inputting a virtual target domain sample into the feature extractor to obtain virtual target domain features; Fitting the classifier using the virtual source domain features and the real virtual source domain labels, and using the fitted classifier to assign virtual target domain neighbor pseudo labels to the virtual target domain samples; Calculate the category prototype based on the virtual target domain neighbor pseudo-label and the virtual target domain feature, and obtain the category prediction of the meta-source model in the virtual target domain sample by matching the category prototype; The neighbor pseudo-label is used as a supervisory signal, and the meta-source model is trained during testing using a cross entropy loss to obtain a maximum posterior probability value of the posterior distribution of the meta-source model in the virtual target domain.

5. The hyperspectral single-source domain generalization method based on test-time training according to claim 4, characterized in that: During training at test time, the cross entropy loss of the source model in the target domain is calculated using the neighbor pseudo-labels to update the meta-source model parameters. The expression of the maximum posterior probability value is as follows: in, is the maximum a posteriori probability value, Φ s is the source model parameter, x t is a sample of the target domain, is the predicted label of the target domain, λ4 is the learning rate of training during testing, is the K nearest neighbor set of the source domain, is the i-th target sample of the K-th category, represents the gradient, L CE (·) is the cross entropy loss, P t is the category prototype of the target domain.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: in, When the program is executed by a processor, the steps of the hyperspectral single-source domain generalization method based on test-time training as described in any one of claims 1 to 5 are implemented.

Images