Cross-domain image classification based on self-supervised spectral space modeling
By employing self-supervised spectral space modeling and diffusion-aligned fine-tuning distillation mechanisms, the problems of domain offset and insufficient information utilization in cross-domain image classification are solved, achieving efficient improvement in cross-domain image classification accuracy and feature extraction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QUANZHOU INST OF EQUIP MFG
- Filing Date
- 2026-03-13
- Publication Date
- 2026-05-12
AI Technical Summary
Cross-domain image classification suffers from domain offset, which causes a sharp drop in model performance in the target domain. Existing methods are inefficient when target domain data cannot be obtained in advance or processed in real time. Furthermore, model complexity and scarce annotations lead to performance degradation, insufficient utilization of spectral and spatial information, and inadequate feature discrimination.
We employ self-supervised spectral spatial modeling, perform dimensionality reduction and serialization through principal component analysis, construct spatial mask features and spectral features, pre-train the spatial-spectral Transformer model, introduce a diffusion alignment fine-tuning distillation mechanism, combine frequency domain constraints and bidirectional cross attention for feature fusion, design a diffusion trajectory aggregation loss, and optimize the student model.
It achieves lightweight adaptation with a small number of target samples, improves the accuracy and discriminative ability of cross-domain image classification, alleviates semantic drift, and enhances the model's cross-domain adaptability and feature extraction ability.
Smart Images
Figure CN121837940B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing, and more specifically to a cross-domain image classification method based on self-supervised spectral spatial modeling. Background Technology
[0002] Cross-domain image classification aims to enable a classification model trained on source domain data, such as data from a specific region or sensor, to be directly and effectively applied to target domain data, such as data from a new region or sensor. This allows for the classification of data collected from different scenes, times, and devices, avoiding the expensive and time-consuming re-labeling and model reconstruction of the target domain data. Its core challenge is "domain shift," which occurs when the same ground features exhibit significant differences in spectral characteristics due to variations in imaging conditions, lighting, season, geographical differences, or sensor physical characteristics, causing a sharp drop in the performance of the source domain model in the target domain.
[0003] Existing methods have the following drawbacks: Domain adaptation methods usually rely on complex joint training of source and target domain data, which is not applicable when the target domain data cannot be obtained in advance or real-time processing is required, resulting in low data efficiency; In order to improve performance, the model tends to be complex, hyperspectral data annotation is scarce, small samples are difficult to support the training of large models, and overfitting is easy.
[0004] The mainstream approach is to learn domain-invariant features to align distributions. However, simply aligning the whole will blur the boundaries between different categories in the feature space, leading to a decrease in inter-class separability. Moreover, in real-world scenarios, the target domain often contains unknown categories, and aligning the whole will lead to "negative transfer," resulting in a performance degradation.
[0005] Most methods fail to fully and synergistically utilize both spectral and spatial information, resulting in insufficient discriminative power of the learned features. Summary of the Invention
[0006] The purpose of this invention is to provide a cross-domain image classification method based on self-supervised spectral space modeling to improve the accuracy of cross-domain image classification.
[0007] To achieve the above objectives, the present invention adopts the following technical solution:
[0008] A cross-domain image classification method based on self-supervised spectral spatial modeling includes the following steps performed sequentially:
[0009] S1: In the data preprocessing stage: acquire the source domain hyperspectral image, use principal component analysis to perform dimensionality reduction and serialization processing on the source domain hyperspectral image, construct spatial mask features and spectral features respectively, and add noise perturbation based on Markov chain to the spatial mask features;
[0010] S2: In the pre-training stage: the spatial-spectral Transformer model is pre-trained using the pre-processed source domain hyperspectral image to obtain the trained spatial-spectral Transformer model;
[0011] The spatial-spectral Transformer model employs a four-layer conditional Transformer encoder and a four-layer Transformer decoder. The first two layers of the encoder use parallel spatial-spectral dual branches to capture local spatial background and spectral distribution features respectively. The middle layer of the encoder uses a bidirectional cross-attention mechanism to achieve deep cross-modal complementarity and obtain fused features. The last two layers of the encoder extract features from the fused features to obtain modeling features. The Transformer decoder denoises and reconstructs the noise features of the modeling features.
[0012] S3: In the fine-tuning stage: the trained spatial-spectral Transformer model is used as the teacher feature network to construct a student model with the same structure and updatable parameters. A diffusion-aligned fine-tuning distillation mechanism is introduced to train the student model and obtain the trained student model.
[0013] S4: During the testing phase: preprocess the target domain HSI image patch, input the preprocessed target domain HSI image patch into the trained student model for image classification processing, and classify the HSI image patch based on the feature representation corresponding to the category label in the model output sequence.
[0014] Preferably, in step S1, principal component analysis is used for dimensionality reduction and serialization to obtain the image. , to image Divided into Image blocks of a certain size, rearranged into a marker sequence. Used for mini-batch training, where , and They represent Height, width, and spectral dimensions, Indicates batch size. Indicates the number of tags, each tag One Image blocks, This indicates the matrix transpose.
[0015] Preferably, in step S2, the spatial feature sequence of the space-spectral Transformer model is constructed by masking each marker in the marker sequence according to a preset masking ratio, thereby dividing the marker sequence into visible parts. and mask part Encode the diffusion time and spatial embedding matrix Add to visible section In the process, spatial feature sequences are obtained. , and These represent the number of visible markers and the feature dimension, respectively.
[0016] Spatial feature sequence Spatial labels are obtained by performing hierarchical feature transformations using two conditional Transformer encoders. ;
[0017] Spectral feature sequence construction: Processing the sequence using one-dimensional convolution along the spectral dimension To extract spectral features The spectral characteristics of each marker are obtained through linear projection. Mapping to the target feature space to obtain projected spectral features Encode the diffusion time Add to this projected spectral feature In the process, spectral feature sequences are obtained. This spectral characteristic sequence The spectral label is obtained after transformation by two conditional Transformer encoders. ;
[0018] A bidirectional cross-attention mechanism is employed for dual cross-modal and cross-modal fusion. The specific operational steps for the spectral-spatial channels are as follows:
[0019] Calculate the spectral label average :
[0020] ;
[0021] Forming cross-attention input:
[0022] ;
[0023] in, Represents the projection matrix. The dimension of the attention head;
[0024] Calculated frequency-guided spatial attention for:
[0025] ;
[0026] in, This indicates that each row of the matrix is normalized;
[0027] Spatial attention is expressed by the following formula. With this spectral label Fusion residuals:
[0028] ;
[0029] The specific operation of the spatial-spectral channel is as follows:
[0030] Computational spatial labeling average :
[0031] ;
[0032] Constructing spectral attention input:
[0033] ;
[0034] in, Represents the projection matrix;
[0035] The frequency attention guided by space is:
[0036] ;
[0037] The spectral label The fusion residual of the frequency attention guided by this space is expressed as:
[0038] ;
[0039] spatial features and spectral characteristics The connections are made, and a linear transformation is applied to obtain the final fused feature representation:
[0040] ;
[0041] in, Input features are transformed into target features through a linear mapping. Multiple features are concatenated along a specified dimension to form a new feature representation;
[0042] This fusion feature The features are then refined by two conditional Transformer encoders, layer normalization (LN), and linear layers to output decoded features.
[0043] Preferably, in step S2, the mask reconstruction stage of the space-spectral Transformer model: a two-layer Transformer decoder is used to perform mask reconstruction on the decoded features, and a frequency domain constraint module is used to perform a real-number fast Fourier transform on the mask reconstruction result. The specific operations are as follows:
[0044] The decoder in the mask reconstruction process places the mask in a visible location. Place learnable markers in the shielded area To initialize the tag sequence, embed the shared location. Add to all tags to form Using a two-layer Transformer decoder to Process the data to generate the final output. Reconstructed features are extracted from the shielded location. ;
[0045] The following formula is used for the mask part and the reconstructed features Perform a real-number fast Fourier transform:
[0046] ;
[0047] in, It is a set of mask tags. and They represent the first time. The ground truth and reconstructed values at each mask location, and the rFFT operation. The dimension is obtained along the channel dimension. The complex spectrum;
[0048] A frequency band mask is used to highlight the high-frequency components:
[0049] ;
[0050] in, Determines the starting value of the high-frequency band (usually taken as...). ), Indicates the frequency channel index. High-frequency components are selected for loss calculation. Exclude low-frequency channels.
[0051] Preferably, the specific steps of the fine-tuning stage in step S3 are as follows:
[0052] The input samples of the diffusion alignment fine-tuning module are divided into a label set. Using diffused noise to target the set of markers The perturbation is represented by the following formula:
[0053] ;
[0054] in, , Represents the cumulative retention factor. Represents standard Gaussian noise, with a random time step. ,in Indicates the total diffusion step size;
[0055] At each time step Teacher Model and student model Output the following class tags respectively:
[0056] ;
[0057] in, and It is the time step Class markers for the teacher and student models;
[0058] Pre-training loss function: During the pre-training phase, the encoder is mainly optimized using two self-supervised objective functions: signal-guided classification loss and diffusion loss. The specific calculation steps for the loss function during the pre-training phase are as follows:
[0059] The signal-guided classification loss corresponds to the frequency-domain constrained FDC module, which consists of a spatial reconstruction term and a frequency-domain constraint term, guiding the reconstruction of the encoder's spatial structure and spectral features.
[0060] This spatial reconstruction uses a pixel-by-pixel approach. loss function :
[0061] ;
[0062] in, express loss;
[0063] The frequency domain constraint uses the high-frequency loss function shown in the following formula. :
[0064] ;
[0065] in, and These represent the spectral amplitude and element-wise multiplication, respectively.
[0066] Reconstruction loss The function is represented by the following formula:
[0067] ;
[0068] Among them, hyperparameters Weights for frequency domain monitoring;
[0069] The diffusion loss: A four-layer diffusion decoder processes this loss through skip connections with the encoder. Output the visual features after noise reduction By targeting cleanliness The mean squared error loss was compared and evaluated:
[0070] ;
[0071] in, This is the mean square error loss;
[0072] The pre-training loss can be expressed as:
[0073] ;
[0074] Fine-tuning loss: The goal of the fine-tuning stage is to enhance the two loss functions, classification loss and diffusion trajectory clustering loss, by utilizing the following signal-to-noise ratio.
[0075] Signal-to-noise ratio enhancement classification loss :
[0076] ;
[0077] in, The class marker representing the student model. This represents the class label obtained from the diffusion perturbation input, the target label. It is a truth value. This represents the noise level measured for each sample and is used for weighted loss;
[0078] Clustering loss of diffusion trajectory: The feature consistency loss at each time step is defined as... :
[0079] ;
[0080] in and These represent time steps of 1 and 2 respectively. Characteristics of the student model and the teacher model at that time It is an optional linear projection, only when and It is only used when the dimensions are inconsistent. Indicates cosine similarity;
[0081] Using clustering loss to force the entire time series Globally consistent alignment of feature trajectories:
[0082] ;
[0083] The fine-tuning loss is:
[0084] ;
[0085] in, It balances alignment and categorization.
[0086] By adopting the aforementioned design scheme, the beneficial effects of the present invention are: during the pre-training period, the present application combines mask modeling with frequency domain awareness, enabling it to learn a transferable spectral-spatial representation, perform lightweight adaptation with minimal target samples, and solve the problems of insufficient target training samples and severe domain offset in hyperspectral cross-domain classification based on deep learning.
[0087] The spatial-spectral Transformer model is designed, which integrates spatial masking and spectral guidance. It performs collaborative spectral-spatial modeling through bidirectional cross-attention, which solves the problems of unbalanced information extraction, insufficient fusion, and difficulty in extracting discriminative features in hyperspectral image classification.
[0088] By introducing frequency domain constraints and using real-number fast Fourier transform and frequency band masking to construct high-frequency mask loss, the model's ability to capture spectral details and its cross-domain discrimination capability are enhanced.
[0089] A diffusion alignment fine-tuning distillation mechanism is proposed, and a diffusion trajectory aggregation loss is constructed to guide the student model to learn the semantic evolution path of the teacher model in the target domain, thereby mitigating semantic drift. Attached Figure Description
[0090] Figure 1 This is a framework diagram of the cross-domain image classification method of the present invention;
[0091] Figure 2 This is a schematic diagram of the structure of the conditional Transformer encoder of the present invention;
[0092] Figure 3 This is a schematic diagram of the bidirectional cross-attention module of the present invention. Detailed Implementation
[0093] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0094] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.
[0095] A cross-domain image classification method based on self-supervised spectral spatial modeling includes the following steps performed sequentially:
[0096] S1: In the data preprocessing stage: acquire the source domain hyperspectral image (HSI), use principal component analysis (PCA) to perform dimensionality reduction and serialization processing on the source domain hyperspectral image, construct spatial mask features and spectral features respectively, and add noise perturbation based on Markov chain to the spatial mask features;
[0097] like Figure 1 As shown, the image was obtained by dimensionality reduction and serialization using principal component analysis. , to image Divided into Image patches of varying sizes are rearranged into a marker sequence. Used for mini-batch training, where , and They represent Height, width, and spectral dimensions, Indicates batch size. Indicates the number of tags, each tag One Image blocks, This indicates the matrix transpose.
[0098] S2: In the pre-training stage: the Spatial-Spectral Transformer (S2Former) model is pre-trained using the pre-processed source domain hyperspectral image to obtain the trained Spatial-Spectral Transformer model;
[0099] like Figure 2As shown, the spatial-spectral Transformer model employs a four-layer Conditional Transformer encoder and a four-layer Transformer decoder. The first two layers of the encoder use parallel spatial-spectral dual branches to capture local spatial background and spectral distribution features respectively. The middle layer of the encoder uses a bidirectional cross-attention mechanism to achieve deep cross-modal complementarity and obtain fused features. The last two layers of the encoder extract features from the fused features to obtain modeling features. The Transformer decoder denoises and reconstructs the noise features of the modeling features.
[0100] In this embodiment, the specific operation steps of the spatial-spectral Transformer model are as follows:
[0101] Spatial feature sequence construction: Each marker in the marker sequence is masked according to a preset mask ratio. In this embodiment, the preset mask ratio is 75%, dividing the marker sequence into visible parts. and mask part Encode the diffusion time and spatial embedding matrix Add to In the process, spatial feature sequences are obtained. It is expressed by the following formula (1):
[0102] (1);
[0103] in, and These represent the number of visible markers and the feature dimension, respectively.
[0104] Spatial feature sequence After undergoing hierarchical feature transformation using two conditional Transformer encoders, the result is expressed by the following formula (2):
[0105] (2);
[0106] in, This represents the operation of a conditional Transformer. Indicates the output of the current layer. This represents the output of the previous layer; spatial labels are obtained by performing hierarchical feature transformations through two conditional Transformer encoders. .
[0107] Spectral feature sequence construction: Processing the labeled sequence using one-dimensional convolution along the spectral dimension To extract spectral features It is expressed by the following formula (3):
[0108] (3);
[0109] in, Indicates size is convolution kernel, These are spectral features obtained through convolution;
[0110] The spectral characteristics of each marker are obtained through linear projection. Mapping to the target feature space to obtain projected spectral features It is expressed by the following formula (4):
[0111] (4);
[0112] in, ;
[0113] Encoding diffusion time Add to this projected spectral feature In the process, spectral feature sequences are obtained. It is expressed by the following formula (5):
[0114] (5);
[0115] Spectral feature sequence The spectral label is obtained after transformation by two conditional Transformer encoders. .
[0116] like Figure 3 As shown, a bidirectional cross-attention mechanism is used for dual cross-modal fusion. The specific operation steps of the spectral-spatial channel are as follows:
[0117] Calculate spectral markers average :
[0118] ;
[0119] Forming cross-attention input:
[0120] ;
[0121] in, Represents the projection matrix. For the dimension of attention head.
[0122] Calculated frequency-guided spatial attention for:
[0123] ;
[0124] in, This means normalizing each row of the matrix so that the sum of the elements in each row is 1.
[0125] Spatial attention is expressed by the following formula. With spectral labeling Fusion residuals:
[0126] ;
[0127] This process guides the spatial branch to focus on regions sensitive to spectral changes through spectral priors, thereby achieving complementary enhancement of spectral and spatial information.
[0128] The specific operations of the spatial-spectral channels are similar to those of the spectral-spatial channels, using uniform pooling to aggregate the spatial representation:
[0129] Computational spatial labeling average :
[0130] ;
[0131] Constructing spectral attention input:
[0132] ;
[0133] in, This represents the projection matrix.
[0134] Therefore, the spatially guided frequency attention is obtained as follows:
[0135] ;
[0136] The spectral label The fusion residual of the frequency attention guided by this space is expressed as:
[0137] ;
[0138] This mechanism enables spectral branches to leverage spatially prominent regions to provide stronger contextual priors, thereby enhancing the discriminative representation of the spectral domain.
[0139] After cross-attention update, spatial features and spectral characteristics The connections are made, and a linear transformation is applied to obtain the final fused feature representation:
[0140] ;
[0141] in, Input features are transformed into target features through a linear mapping. Multiple features are concatenated along a specified dimension to form a new feature representation;
[0142] This fusion feature The features are then refined by two conditional Transformer encoders, layer normalization (LN), and linear layers to output decoded features.
[0143] In the mask reconstruction stage: a two-layer Transformer decoder is used to perform mask reconstruction on the decoded features. A frequency domain constraint (FDC) module is used to perform a real fast Fourier transform (rFFT) on the mask reconstruction result, and a high-frequency loss function is constructed by combining the frequency band mask. .
[0144] In this embodiment, to improve the model's ability to capture fine-grained spectral features in hyperspectral images, this application introduces a frequency-domain constraint (FDC) module in the self-supervised pre-training stage, such as... Figure 1 As shown.
[0145] The decoder in the mask reconstruction process places the mask in a visible location. Place learnable markers in the shielded area To initialize the sequence. Embed shared locations. Add to all tags to form Using a two-layer Transformer decoder to Process the data to generate the final output. Reconstructed features are extracted from the shielded location. .
[0146] To evaluate the quality of the spectral signal recovery in the frequency domain, the following formula is used for the mask portion. and reconstructed features Perform real-number fast Fourier transform (rFFT):
[0147] ;
[0148] in, It is a set of mask tags. and They represent the first time. The ground truth and reconstructed values at each mask location. rFFT operation. The dimension is obtained along the channel dimension. The complex spectrum.
[0149] This invention uses a frequency band mask to highlight the high-frequency components:
[0150] ;
[0151] in, Determines the starting value of the high-frequency band (usually taken as...). ), Indicates the frequency channel index. High-frequency components are selected for loss calculation. Low-frequency channels are excluded. By introducing a frequency band masking mechanism, the model can focus on the recovery of high-frequency information, avoiding the impact of low-frequency information on the frequency domain loss calculation.
[0152] S3: In the fine-tuning stage: Based on the target domain data, the data preprocessing process of step S1 is executed, and the encoder and intermediate feature extraction layer obtained in the pre-training stage are frozen as the teacher feature network. At the same time, a student model with the same structure and updatable parameters is constructed. A diffusion alignment fine-tuning distillation mechanism is introduced to train the student model and obtain the trained student model.
[0153] A Diffusion-Aligned Fine-Tuning (DAFT) distillation mechanism is introduced, which aligns the semantic evolution trajectories at different time steps during the diffusion process through a teacher-student structure. The classification loss (loss_cls) is calculated using labeled samples in the target domain, and the consistency between the teacher and student in the diffusion semantic space is constrained by the diffusion trajectory aggregation loss (loss_dta). The diffusion-aligned fine-tuning distillation mechanism adopts a three-way joint loss consisting of classification supervision, time step alignment, and distillation consistency objective, thereby achieving stable and efficient cross-domain transfer optimization in the few-sample fine-tuning process.
[0154] The input samples of the diffusion alignment fine-tuning module are divided into a label set. Using diffused noise to target the set of markers The perturbation is represented by the following formula:
[0155] ;
[0156] in, , Represents the cumulative retention factor. Represents standard Gaussian noise, with a random time step. ,in This indicates the total diffusion step size.
[0157] At each time step Teacher Model and student model Output the following class tags respectively:
[0158] ;
[0159] in, and It is the time step The class labels of the teacher and student models are used to measure their feature consistency within the diffuse semantic space.
[0160] To enhance the model's generalization and cross-domain adaptability, this application designs multiple loss functions for joint optimization during the pre-training and fine-tuning stages.
[0161] Pre-training loss function: During the pre-training phase, the encoder is mainly optimized using two self-supervised objective functions: signal-guided classification loss and diffusion loss (DFS). The specific calculation steps for the loss function during the pre-training phase are as follows:
[0162] The signal-guided classification loss corresponds to the frequency-domain constrained FDC module, which consists of a spatial reconstruction term and a frequency-domain constraint term, guiding the reconstruction of the encoder's spatial structure and spectral features.
[0163] Spatial reconstruction items use pixel-by-pixel loss function :
[0164] ;
[0165] in, express loss;
[0166] The frequency domain constraint term uses the high-frequency loss function shown in the following formula. :
[0167] ;
[0168] in, and These represent the spectral amplitude and element-wise multiplication, respectively. This constraint enables the model to maintain the accurate distribution of the high-frequency region, thereby improving its ability to perceive fine-grained spectral differences.
[0169] Reconstruction loss The function is represented by the following formula:
[0170] ;
[0171] Among them, hyperparameters The weights for frequency domain supervision are typically set to 0.5; by weighting and fusing the two losses mentioned above, the... and Collaborative constraints optimize the spatial and frequency features of images, thereby achieving higher quality detail and structure restoration in reconstruction tasks.
[0172] Diffusion Loss (DFS) function: such as Figure 1 As shown, a four-layer diffusion decoder processes data via skip connections with the encoder. Output the visual features after noise reduction By targeting cleanliness The mean squared error loss was compared and evaluated:
[0173] ;
[0174] in, This is the mean square error loss;
[0175] pass Measure the difference between the denoised image and the real image, and optimize the model's ability to gradually remove noise and restore image details during the diffusion process;
[0176] Overall, the pre-training loss can be expressed as:
[0177] ;
[0178] The pre-training loss is used to back-adjust the spatial-spectral Transformer model during the pre-training phase. During training, the weights of the two loss terms are equal, guiding the model to recover spectral-spatial structure information at the mask location and enhancing robust modeling against perturbations in the visible region.
[0179] Fine-tuning loss: The goal of the fine-tuning stage is to enhance the two loss functions, classification loss and diffusion trajectory clustering loss, using the following signal-to-noise ratio to achieve strong discriminative ability and source-target consistency in the target domain.
[0180] Signal-to-noise ratio (SNR) enhancement classification loss: To enhance the model's stability under different SNR levels after diffusion perturbation, a SNR enhancement classification loss is employed. :
[0181] ;
[0182] in, The class marker representing the student model. This represents the class label obtained from the diffusion perturbation input, the target label. It is a truth value. The noise level of each sample is measured and used for weighted loss, where low-noise samples (high signal-to-noise ratio) contribute less to the total loss, while high-noise samples (low signal-to-noise ratio) contribute more.
[0183] Diffusion Trajectory Clustering Loss: The student model learns the dynamic representation trajectory of the teacher model across multiple diffusion time steps. This mechanism aligns semantic trajectories through knowledge distillation. To achieve semantic alignment across time steps, this application defines the feature consistency loss for each time step as follows: :
[0184] ;
[0185] in and These represent time steps of 1 and 2 respectively. Characteristics of the student model and the teacher model; It is an optional linear projection, only when and It is only used when the dimensions are inconsistent; This represents the cosine similarity.
[0186] Cluster loss is introduced to force the entire time series Globally consistent alignment of feature trajectories:
[0187] ;
[0188] Overall, the total fine-tuning loss is:
[0189] ;
[0190] in, Alignment and classification are balanced to ensure that student models adapt to the target domain while preserving semantic knowledge of the source domain.
[0191] S4: During the testing phase: PCA dimensionality reduction and serialization are performed on the target domain HSI image patch, and it is directly input into the trained student model backbone network; finally, the HSI image patch is classified based on the feature representation corresponding to the category label in the model output sequence.
[0192] In summary, this application proposes a self-supervised cross-domain classification framework that combines mask modeling with frequency domain awareness during pre-training, enabling it to learn a transferable spectral-spatial representation and perform lightweight adaptation with minimal target samples. This addresses the problems of insufficient target training samples and severe domain shift in deep learning-based hyperspectral cross-domain classification.
[0193] The spatial-spectral Transformer model is designed, which integrates spatial masking and spectral guidance. It performs collaborative spectral-spatial modeling through bidirectional cross-attention, which solves the problems of unbalanced information extraction, insufficient fusion, and difficulty in extracting discriminative features in hyperspectral image classification.
[0194] By introducing frequency domain constraints and using real-number fast Fourier transform and frequency band masking to construct high-frequency mask loss, the model's ability to capture spectral details and its cross-domain discrimination capability are enhanced.
[0195] A diffusion alignment fine-tuning distillation mechanism is proposed, and a diffusion trajectory aggregation loss is constructed to guide the student model to learn the semantic evolution path of the teacher model in the target domain, thereby mitigating semantic drift.
[0196] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A cross-domain image classification method based on self-supervised spectral spatial modeling, characterized in that: The steps are as follows, performed sequentially: S1: In the data preprocessing stage: acquire the source domain hyperspectral image, use principal component analysis to perform dimensionality reduction and serialization processing on the source domain hyperspectral image, construct spatial mask features and spectral features respectively, and add noise perturbation based on Markov chain to the spatial mask features; S2: In the pre-training stage: the spatial-spectral Transformer model is pre-trained using the pre-processed source domain hyperspectral image to obtain the trained spatial-spectral Transformer model; The spatial-spectral Transformer model employs a four-layer conditional Transformer encoder and a four-layer Transformer decoder. The first two layers of the encoder use parallel spatial-spectral dual branches to capture local spatial background and spectral distribution features respectively. The middle layer of the encoder uses a bidirectional cross-attention mechanism to achieve deep cross-modal complementarity and obtain fused features. The last two layers of the encoder extract features from the fused features to obtain modeling features. The Transformer decoder denoises and reconstructs the noise features of the modeling features. S3: In the fine-tuning stage: the trained spatial-spectral Transformer model is used as the teacher feature network to construct a student model with the same structure and updatable parameters. A diffusion-aligned fine-tuning distillation mechanism is introduced to train the student model and obtain the trained student model. S4: During the testing phase: preprocess the target domain HSI image patch, input the preprocessed target domain HSI image patch into the trained student model for image classification processing, and classify the HSI image patch based on the feature representation corresponding to the category label in the model output sequence.
2. The cross-domain image classification based on self-supervised spectral spatial modeling as described in claim 1, characterized in that: In step S1, principal component analysis is used for dimensionality reduction and serialization to obtain the image. , to image Divided into Image blocks of a certain size, rearranged into a marker sequence. Used for mini-batch training, where , and They represent Height, width, and spectral dimensions, Indicates batch size. Indicates the number of tags, each tag One Image blocks, This indicates the matrix transpose.
3. The cross-domain image classification based on self-supervised spectral spatial modeling as described in claim 2, characterized in that: In step S2, the spatial feature sequence of the space-spectral Transformer model is constructed by masking each marker in the marker sequence according to a preset masking ratio, thus dividing the marker sequence into visible parts. and mask part Encode the diffusion time and spatial embedding matrix Add to visible section In the process, spatial feature sequences are obtained. , and These represent the number of visible markers and the feature dimension, respectively. Spatial feature sequence Spatial labels are obtained by performing hierarchical feature transformations using two conditional Transformer encoders. ; Spectral feature sequence construction: Processing the labeled sequence using one-dimensional convolution along the spectral dimension To extract spectral features The spectral characteristics of each marker are obtained through linear projection. Mapping to the target feature space to obtain projected spectral features Encode the diffusion time Add to this projected spectral feature In the process, spectral feature sequences are obtained. This spectral characteristic sequence The spectral label is obtained after transformation by two conditional Transformer encoders. ; A bidirectional cross-attention mechanism is employed for dual cross-modal and cross-modal fusion. The specific operational steps for the spectral-spatial channels are as follows: Calculate the spectral label average : ； Forming cross-attention input: ； in, Represents the projection matrix. The dimension of the attention head; Calculated frequency-guided spatial attention for: ； in, This indicates that each row of the matrix is normalized; Spatial attention is expressed by the following formula. With this spectral label Fusion residuals: ； The specific operation of the spatial-spectral channel is as follows: Computational spatial labeling average : ； Constructing spectral attention input: ； in, Represents the projection matrix; The frequency attention guided by space is: ； The spectral label The fusion residual of the frequency attention guided by this space is expressed as: ； spatial features and spectral characteristics The connections are made, and a linear transformation is applied to obtain the final fused feature representation: ； in, Input features are transformed into target features through a linear mapping. Multiple features are concatenated along a specified dimension to form a new feature representation; This fusion feature The features are then refined by two conditional Transformer encoders, layer normalization (LN), and linear layers to output decoded features.
4. The cross-domain image classification based on self-supervised spectral spatial modeling as described in claim 3, characterized in that: In step S2, the mask reconstruction stage of the space-spectral Transformer model: a two-layer Transformer decoder is used to perform mask reconstruction on the decoded features, and a frequency domain constraint module is used to perform a real-number fast Fourier transform on the mask reconstruction result. The specific operations are as follows: The decoder in the mask reconstruction process places the mask in a visible location. Place learnable markers in the shielded area To initialize the tag sequence, embed the shared location. Add to all tags to form Using a two-layer Transformer decoder to Process the data to generate the final output. Reconstructed features are extracted from the shielded location. ; The following formula is used for the mask part and the reconstructed features Perform a real-number fast Fourier transform: ； in, It is a set of mask tags. and They represent the first time. The ground truth and reconstructed values at each mask location, and the rFFT operation. The dimension is obtained along the channel dimension. The complex spectrum; A frequency band mask is used to highlight the high-frequency components: ； in, It determines the starting value of the high-frequency band. Indicates the frequency channel index. High-frequency components are selected for loss calculation. Exclude low-frequency channels.
5. The cross-domain image classification based on self-supervised spectral spatial modeling as described in claim 4, characterized in that: The specific steps for the fine-tuning stage in step S3 are as follows: The input samples of the diffusion alignment fine-tuning module are divided into a label set. Using diffused noise to target the set of markers The perturbation is represented by the following formula: ； in, , Represents the cumulative retention factor. Represents standard Gaussian noise, with a random time step. ,in Indicates the total diffusion step size; At each time step Teacher Model and student model Output the following class tags respectively: ； in, and It is the time step Class markers for the teacher and student models; Pre-training loss function: During the pre-training phase, the encoder is mainly optimized using two self-supervised objective functions: signal-guided classification loss and diffusion loss. The specific calculation steps for the loss function during the pre-training phase are as follows: The signal-guided classification loss corresponds to the frequency-domain constrained FDC module, which consists of a spatial reconstruction term and a frequency-domain constraint term, guiding the reconstruction of the encoder's spatial structure and spectral features. This spatial reconstruction uses a pixel-by-pixel approach. loss function : ； in, express loss; The frequency domain constraint uses the high-frequency loss function shown in the following formula. : ； in, and These represent the spectral amplitude and element-wise multiplication, respectively. Reconstruction loss The function is represented by the following formula: ； Among them, hyperparameters Weights for frequency domain monitoring; The diffusion loss: A four-layer diffusion decoder processes this loss through skip connections with the encoder. Output the visual features after noise reduction By working with clean targets The mean squared error loss was compared and evaluated: ； in, This is the mean square error loss; The pre-training loss can be expressed as: ； Fine-tuning loss: The goal of the fine-tuning stage is to enhance the two loss functions, classification loss and diffusion trajectory clustering loss, by utilizing the following signal-to-noise ratio. Signal-to-noise ratio enhancement classification loss : ； in, The class marker representing the student model. This represents the class label obtained from the diffusion perturbation input, the target label. It is a truth value. This represents the noise level measured for each sample and is used for weighted loss; Clustering loss of diffusion trajectory: The feature consistency loss at each time step is defined as... : ； in and These represent time steps of 1 and 2 respectively. Characteristics of the student model and the teacher model at that time It is an optional linear projection, only when and It is only used when the dimensions are inconsistent. Indicates cosine similarity; Using cluster loss to force the entire time series Globally consistent alignment of feature trajectories: ； The fine-tuning loss is: ； in, It balances alignment and categorization.