A test-time trained hyperspectral single-source domain generalization method and storage medium

Abstract

This application discloses a hyperspectral single-source domain generalization method and storage medium based on test-time training. Multiple virtual source domains are constructed by randomizing the original hyperspectral image data domain. Domain-invariant knowledge is learned by aligning the Hessian matrices of the virtual source domains, and a loss function is determined accordingly to train the meta-source model. During meta-target adaptation, the maximum a posteriori probability value of the observation dynamic inference model for the virtual target domain is obtained through variational posterior inference based on virtual target domain samples and nearest-neighbor pseudo-labels. In meta-target testing, based on the maximum a posteriori probability value constraint, the log-likelihood between virtual target domain samples and true labels is maximized to optimize the meta-source model. The meta-target loss is derived to determine the update rule, which is evaluated in the virtual target domain and guides optimization. During the meta-target testing phase, nearest-neighbor pseudo-labels are assigned to the test target domain samples to obtain the predicted category of the actual target domain samples. This application improves the cross-domain generalization ability of the model under the condition that the target domain data source domain is inaccessible during training.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and in particular relates to a hyperspectral single-source domain generalization method and storage medium based on test-time training. Background Technology

[0002] In recent years, deep learning-based hyperspectral image (HSI) classification methods have demonstrated superior performance in specific tasks. However, the success of these methods often relies on the ideal assumption that the training data (source domain) and test data (target domain) are identically distributed. This assumption is difficult to uphold in practical applications. Due to factors such as acquisition time, geographical location, sensor differences, and environmental factors, significant distributional biases often exist between different HSI data sets. Especially when the distribution differences between the source and target domains are large, models trained on the source domain typically perform poorly in the target domain. In many real-world scenarios, models need to handle classification tasks for regions not observed during training.

[0003] Therefore, how to improve the cross-domain generalization ability of the model under the condition that the target domain data source domain is inaccessible during training has become a key problem that needs to be solved in the field of HSI classification. Summary of the Invention

[0004] The technical objective of this application is to address the problem of how to improve the cross-domain generalization ability of a model when the target domain data source is inaccessible during training.

[0005] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0006] In a first aspect, embodiments of this application provide a hyperspectral single-source domain generalization method based on test-time training, comprising the following steps:

[0007] Step 1: Perform domain randomization on the original hyperspectral image data to construct multiple virtual source domains;

[0008] Step 2: Learn domain-invariant knowledge by aligning the Hessian matrices of each virtual source domain, determine the domain-invariant loss function based on the domain-invariant knowledge, and train the meta-source model based on the domain-invariant loss function;

[0009] Step 3, perform the meta-target adaptation process: The meta-source model, based on virtual target domain samples and corresponding nearest neighbor pseudo-labels, determines the observation dynamics of the virtual target domain through variational posterior inference and then infers the maximum posterior probability value of the meta-source model in the virtual target domain.

[0010] Step 4, Perform the meta-target testing process: Based on the constraint of the maximum posterior probability value, optimize the meta-source model by maximizing the log-likelihood between the virtual target domain samples and the real labels under the condition of the meta-source model parameters; based on the derived meta-target loss, determine the meta-source model update rule in the meta-target testing stage, evaluate the meta-source model loss in the virtual target domain, and guide the optimization of the meta-source model.

[0011] Step 5, Meta-target testing phase: Use the meta-source model to assign nearest neighbor pseudo-labels to the test target domain samples, and obtain the predicted category corresponding to the actual target domain sample based on the nearest neighbor pseudo-labels.

[0012] Furthermore, the original hyperspectral image data is domain randomized to construct multiple virtual source domains, including:

[0013] By utilizing a combination of spectral perturbation, spectral sampling, spectral smoothing, pixel reconstruction, and spatial rotation, the original hyperspectral image data is subjected to spectral and / or spatial interventions to construct multiple virtual source domains.

[0014] Furthermore, the domain-invariant loss function contains a term of the s-th term. i virtual source domain and the s j virtual source domain A measure of cross-domain knowledge transfer;

[0015] Step 2 specifically includes: optimizing the cross-domain knowledge transfer measure and obtaining the upper bound of the cross-domain knowledge transfer measure;

[0016] Based on the upper bound, the problem of upper bound with a quadratic form is transformed into a simple norm product relationship using the spectral norm, thus obtaining a simplified upper bound; the domain-invariant loss function is updated based on the simplified upper bound; and the metasource model is trained using the updated domain-invariant loss function.

[0017] Virtual source domain features are obtained using the updated feature extractor of the meta-source model.

[0018] Furthermore, the meta-model includes a feature extractor and a classifier; step 3 specifically includes:

[0019] The virtual target domain sample is input into the feature extractor to obtain the virtual target domain features;

[0020] The classifier is fitted with the virtual source domain features and the real labels, and the fitted classifier is used to assign nearest neighbor pseudo labels to the virtual target domain samples.

[0021] The category prototype is calculated based on the nearest neighbor pseudo-label and the virtual target domain features, and the category prediction of the metasource model in the virtual target domain sample is obtained by matching the category prototype.

[0022] Using the nearest neighbor pseudo-labels as supervision signals, the metasource model is trained during testing using cross-entropy loss, thereby obtaining the maximum posterior probability value of the metasource model's posterior distribution in the virtual target domain.

[0023] Furthermore, the expression for the maximum a posteriori probability value is as follows:

[0024]

[0025] in Φ is the maximum posterior probability value. s For the source model parameters, x t For samples in the target domain, Let λ be the predicted label for the target domain, and λ4 be the learning rate during testing. Let K be the set of K nearest neighbors of the source domain. For the i-th target sample of the K-th category, L represents the gradient. CE (·) represents the cross-entropy loss, P t This is the category prototype for the target domain.

[0026] Secondly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, it implements the steps of the test-time trained hyperspectral single-source domain generalization method provided in any possible implementation of the first aspect.

[0027] Compared with existing technologies, the beneficial technical effects of the hyperspectral single-source domain generalization method based on test-time training provided in this application include: firstly, it innovatively proposes a gradient regularization strategy based on the Hessian matrix, aiming to mine potential domain-invariant knowledge from gradient information and improve the model's generalization ability; secondly, it develops a test-time training mechanism based on Bayesian posterior parameter inference, enabling the domain generalization model to dynamically adapt to changes in the data distribution of the target domain, achieving personalized adaptation to the target domain; and thirdly, it integrates gradient regularization and the test-time training mechanism into a meta-learning framework consisting of three stages: meta-source training, meta-target adaptation, and meta-target testing. By simulating the generalization scenario during testing in the source domain training stage, it ensures that the model has robust generalization ability when facing an unseen target domain. Attached Figure Description

[0028] Figure 1 This is a schematic block diagram illustrating the method principle provided in the embodiments of this application. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0030] To address the challenge of enhancing cross-domain generalization capabilities of a model when the target domain data source is inaccessible during training, this application provides a hyperspectral single-source domain generalization method incorporating a Bayesian meta-Hessian network. First, it overcomes the limitations of traditional methods that focus solely on learning domain invariance by innovatively proposing a gradient regularization strategy based on the Hessian matrix. This strategy aims to extract potential domain-invariant knowledge from gradient information, thereby improving the model's generalization ability. Second, a test-time training mechanism based on Bayesian posterior parameter inference is developed, enabling the domain generalization model to dynamically adapt to changes in the target domain's data distribution, achieving personalized adaptation to the target domain. Third, the gradient regularization and test-time training mechanisms are integrated into a meta-learning framework consisting of three stages: meta-source training, meta-target adaptation, and meta-target testing. By simulating the generalization scenario during testing during the source domain training stage, the method ensures robust generalization capabilities when facing an unseen target domain.

[0031] The present application will be further described below with reference to the accompanying drawings.

[0032] like Figure 1 As shown, the embodiment provides a test-time training-based hyperspectral single-source domain generalization method, including:

[0033] Step 1: To simulate domain generalization scenarios under single-source domain conditions, the original hyperspectral image (HSI) data is subjected to diverse spectral and spatial interventions through domain randomization to construct multiple virtual domains, including virtual source domains and virtual target domains, and to simulate the distribution changes between different domains.

[0034] Domain randomization is based on the original source domain D. s Generate virtual source domain s i N is the virtual source domain index. s This represents the number of virtual source domains.

[0035] The domain randomization used in some embodiments includes a combination of multiple methods such as spectral perturbation, spectral sampling, spectral smoothing, pixel recombination, spatial rotation, and pixel reset.

[0036] Spectral perturbation: This operation simulates potential spectral changes caused by environmental noise in cross-domain scenarios by adding noise to the original source domain data.

[0037] Spectral sampling: This translation operation simulates spectral changes caused by differences in spectral resolution between different devices in cross-domain scenarios by sampling the spectral curve of the original source domain data at fixed intervals.

[0038] Spectral smoothing: This operation smooths the spectral curve of the original source domain data by filtering, simulating spectral changes caused by differences in sensor sensitivity in cross-domain scenarios.

[0039] Pixel Reassembly: This operation simulates the distribution changes of ground features with different HSIs in a cross-domain scene by shuffling the spatial order of pixels in the original source domain data image block.

[0040] Spatial rotation: This operation simulates the change in perspective when acquiring different HSIs in a cross-domain scene by rotating the original source domain hyperspectral image patch.

[0041] The method provided in this embodiment constructs multiple virtual source domains to simulate different data distributions, enabling the model to access diverse data features during training without relying on the direct participation of the target domain data source during training. Even if the target domain data source is inaccessible during training, the model can still learn cross-domain knowledge through the virtual source domains.

[0042] Step 2: Based on each virtual source domain, perform meta-source training. The model learns domain-invariant knowledge by aligning the Hessian matrices of different virtual source domains.

[0043] During primitive training, some implementations employ the following domain-invariant loss:

[0044]

[0045] Among them, L D For the domain-invariant loss value, and Let f(·) be the sample from the virtual source domain and c(·) be the label from the real virtual source domain. Let f(·) be the feature extractor and c(·) be the KNN classifier. CE (·) represents the cross-entropy loss. For virtual source domain and The cross-domain knowledge transfer measure, N s It represents the number of virtual source domains.

[0046] By optimizing its upper bound to maximize the cross-domain knowledge transfer measure across different domains. The upper bound can be expressed as:

[0047]

[0048] Where ζ is the δ-minimal set, and π is the classifier parameter. *These are the optimal parameters for the meta-source model in the two virtual source domains. Indicates virtual source domain The Hessian matrix, Indicates virtual source domain The Hessian matrix, o(ζ) 2 ) represents an infinitesimal quantity.

[0049] Next, the problem of upper bounds with quadratic form is transformed into a simple norm product relationship using spectral norms, thus simplifying the calculation of the upper bound.

[0050] The domain-invariant loss can be expressed as:

[0051]

[0052] Next, the meta-model is trained based on domain-invariant loss:

[0053]

[0054] Where, Φ s' For the parameters of the original model, Φ s Here, λ represents the source model parameters, and λ1 represents the learning rate during the source model training phase. Represents the gradient. Indicates the source domain tag. x s This represents a sample from the source domain.

[0055] Finally, the virtual source domain features are obtained using the feature extractor of the updated metasource model.

[0056] The goal of this embodiment is to train a high-performing source model that performs well in the target domain. By simulating test scenarios in the target domain within the source domain, the source model is trained using virtual source domain samples and corresponding virtual source domain pseudo-labels as supervisory signals. The parameters of the source model are defined as source model parameters Φ. s Since there is no supervised signal in the target domain, training a model close to optimal is called a meta-model. The parameters of the meta-model are defined as the meta-model parameters Φ. s' .

[0057] Step 3, Perform the meta-target adaptation process: The meta-source model, based on virtual target domain samples and corresponding virtual target domain nearest neighbor pseudo-labels, calculates the maximum posterior probability value of the target domain meta-source model parameters through variational posterior inference to achieve intelligent adaptation to the target domain.

[0058] First, the virtual target domain features are obtained using the feature extractor of the metasource model. Then, a KNN classifier is fitted using the virtual source domain features and corresponding real virtual target domain labels. Virtual target domain samples are assigned virtual target domain pseudo-labels by searching for their K nearest neighbors within the virtual source domain. Next, a class prototype is calculated based on the virtual target domain pseudo-labels and virtual target domain features, and the model's class prediction in the virtual target domain is obtained through prototype matching. Finally, the virtual target domain pseudo-labels are used as supervision signals, and the metasource model is trained during testing using cross-entropy loss to obtain the model's posterior distribution p(Φ) in the virtual target domain. t' |P t' ,x t' ,Φ s' The maximum posterior probability value The optimization process at this stage can be represented as:

[0059]

[0060] Where λ2 is the learning rate of the meta-objective adaptation process, x t' For virtual target domain samples, P t' For the category prototype of the virtual target domain, For gradient, Let q represent the K nearest neighbor set of the source domain. θ (·) Conditional probability distribution.

[0061] Step 4, Perform the meta-target testing process: In order to further simulate the training during testing of the target domain, some embodiments utilize the maximum posterior probability value learned in the meta-target adaptation task to optimize the meta-source model by maximizing the log-likelihood between the target domain data and the true label under the condition of maximizing the meta-source model parameters.

[0062] As an example, at this stage, the model parameters are supervised by maximizing the following log-likelihood: Meta-target prediction process under the given conditions:

[0063]

[0064] Among them, maximizing the first term Equivalent to minimizing the model parameters as Under the given conditions, the model's cross-entropy loss in the virtual target domain. Minimizing the second term is equivalent to minimizing the KL divergence between the class prototype distribution obtained based on pseudo-labels and the class prototype distribution obtained based on real labels.

[0065] Therefore, the following meta-target loss can be obtained:

[0066]

[0067] The update rule for the model during the meta-objective testing phase can be expressed as:

[0068]

[0069] Where λ3 is the learning rate during the meta-objective testing phase. It should be noted that L... T is based on Calculated, but the update is in Φ s' The optimization of the meta-source model is guided by evaluating the model's loss on a virtual target domain, thereby improving its adaptability to target domains not seen during the meta-target testing phase.

[0070] In this embodiment, the loss function is determined by learning domain-invariant knowledge, and the meta-source model is trained. The optimization during meta-target adaptation and testing is also based on the general knowledge already learned by the model and a small amount of target domain-related information (such as virtual target domain samples and pseudo-labels). The entire process does not depend on the accessibility of the target domain data source during training. Instead, through clever design, the model is gradually adjusted at different stages to adapt to the target domain, thereby improving the model's cross-domain generalization ability when the target domain data source is inaccessible.

[0071] Step 5 proposes a nearest neighbor pseudo-label mechanism, which maps target domain samples to K nearest neighbor samples in the source domain, reducing the generalization risk of the model in the target domain, improving the quality of target domain pseudo-labels during training and testing, and helping the metasource model to more accurately approximate the posterior distribution of the virtual target domain.

[0072] During the meta-target testing phase, the true labels in the target domain are inaccessible. This invention calculates the category prototype based on the nearest neighbor pseudo-labels of the target domain samples, and then obtains the corresponding category prediction through prototype metric. During testing and training, the cross-entropy loss of the source model in the target domain is calculated using the nearest neighbor pseudo-labels to update the parameters of the original model, thereby obtaining... Maximum posterior probability

[0073]

[0074] Where λ4 is the learning rate during testing.

[0075] Finally, based on Complete the prediction for the target domain.

[0076] In this embodiment, the meta-source training phase simulates the model's domain-invariant knowledge learning process in the source domain. Gradient regularization constrains the model to have the same optimization path across different virtual source domains, minimizing the upper bound of the cross-domain knowledge transfer metric. The meta-target adaptation phase simulates the personalized adaptation process in the target domain. Based on Bayesian theory and variational inference, it dynamically infers the posterior parameter distribution of the model in the virtual target domain based on observations. The meta-target testing phase simulates the model's testing process in the target domain. By minimizing the prediction likelihood between virtual target domain samples and labels under the condition of the meta-source model parameters, it guides the optimization of the meta-source model, thereby improving its adaptability to target domains not seen in the meta-target testing phase. Experimental results on multiple HSI datasets demonstrate that the proposed method achieves state-of-the-art performance in hyperspectral generalization tasks.

[0077] This application also provides a computer-readable storage medium storing a computer program thereon, wherein when the program is executed by a processor, it implements the steps of the test-time training-based hyperspectral single-source domain generalization method provided in the above embodiments.

[0078] The above provides a detailed description of the hyperspectral single-source domain generalization method and storage medium based on test-time training provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the concept of this application and should not be construed as limiting the scope of protection of this application.

Claims

1. A test-time training-based hyperspectral single-source domain generalization method, characterized in that, Includes the following steps: Step 1: Perform domain randomization on the original hyperspectral image data to construct multiple virtual source domains; Step 2: Learn domain-invariant knowledge by aligning the Hessian matrices of each virtual source domain, determine the domain-invariant loss function based on the domain-invariant knowledge, and train the meta-source model based on the domain-invariant loss function; Step 3, Meta-target adaptation process: The meta-source model, based on virtual target domain samples and corresponding virtual target domain nearest neighbor pseudo-labels, determines the observation dynamics of the virtual target domain through variational posterior inference and then infers the maximum posterior probability value of the meta-source model in the virtual target domain. Step 4, Perform the meta-target testing process: Based on the constraint of the maximum a posteriori probability value, optimize the meta-source model by maximizing the log-likelihood between the virtual target domain samples and the real labels under the condition of the meta-source model parameters; based on the derived meta-target loss, determine the meta-source model update rule in the meta-target testing stage, evaluate the meta-source model loss in the virtual target domain, and guide the optimization of the meta-source model; Step 5, Meta-target testing phase: Use the meta-source model to assign nearest neighbor pseudo-labels to the test target domain samples, and obtain the predicted category corresponding to the actual target domain sample based on the nearest neighbor pseudo-labels.

2. The hyperspectral single-source domain generalization method based on test-time training according to claim 1, characterized in that, Domain randomization is performed on the original hyperspectral image data to construct multiple virtual source domains, including: By utilizing a combination of spectral perturbation, spectral sampling, spectral smoothing, pixel reconstruction, and spatial rotation, the original hyperspectral image data is subjected to spectral and / or spatial interventions to construct multiple virtual source domains.

3. The hyperspectral single-source domain generalization method based on test-time training according to claim 1, characterized in that, The domain-invariant loss function contains a term of the s-th term. i virtual source domain and the s j virtual source domain A measure of cross-domain knowledge transfer; Step 2 specifically includes: optimizing the cross-domain knowledge transfer measure and obtaining the upper bound of the cross-domain knowledge transfer measure; Based on the aforementioned upper bound, the problem of upper bounds with quadratic forms is transformed into a simple norm product relationship using the spectral norm, thus obtaining a simplified upper bound. The domain-invariant loss function is updated based on the simplified upper bound; the metasource model is trained using the updated domain-invariant loss function. Virtual source domain features are obtained using the updated feature extractor of the meta-source model.

4. The hyperspectral single-source domain generalization method based on test-time training according to claim 1, characterized in that, The meta-model includes a feature extractor and a classifier; step 3 specifically includes: The virtual target domain sample is input into the feature extractor to obtain the virtual target domain features; The classifier is fitted with the virtual source domain features and the real virtual source domain labels, and the fitted classifier is used to assign virtual target domain nearest neighbor pseudo labels to the virtual target domain samples. The category prototype is calculated based on the nearest neighbor pseudo-labels of the virtual target domain and the features of the virtual target domain, and the category prediction of the metasource model in the virtual target domain sample is obtained by matching the category prototypes. Using the nearest neighbor pseudo-labels as supervision signals, the metasource model is trained during testing using cross-entropy loss, thereby obtaining the maximum posterior probability value of the metasource model's posterior distribution in the virtual target domain.

5. The hyperspectral single-source domain generalization method based on test-time training according to claim 4, characterized in that, During training and testing, the cross-entropy loss of the source model in the target domain is calculated using the nearest neighbor pseudo-labels to update the parameters of the source model. The expression for the maximum posterior probability value is as follows: in, Φ is the maximum a posteriori probability value. s For the source model parameters, x t For samples in the target domain, Let λ be the predicted label for the target domain, and λ4 be the learning rate during testing. Let K be the set of K nearest neighbors of the source domain. For the i-th target sample of the K-th category, L represents the gradient. CE (·) represents the cross-entropy loss, P t This is the category prototype for the target domain.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, in, When the program is executed by the processor, it implements the steps of the test-time training-based hyperspectral single-source domain generalization method as described in any one of claims 1-5.

Images