Domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation
Through the three-dimensional frequency domain transformation and intermediate domain data fusion methods, the model performance degradation caused by the inter-domain difference in hyperspectral images across scene classification is solved, and the robustness and generalization ability of the model are improved, especially in multi-source In the domain environment.
Patent Information
- Application Number
- CN202411273536.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-09-12
AI Technical Summary
In the existing cross-scene classification task of hyperspectral images, the main reason for the decline in model performance is inter-domain differences. The existing research is mainly based on single-source domain generalization, ignoring the frequency domain information, which leads to the generalization of the model when facing domain offset. Inadequate ability to transform.
Using the three-dimensional frequency domain transformation method, the source domain image is converted to the frequency domain through Fourier transform, amplitude mixing and phase recombination are performed, extended domain images are generated, and adaptive frequency domain transformation modules and features are constructed through the fusion of intermediate domain data. Extractor, training generator and discriminator, learning domain unchanged representation.
It improves the model's ability to identify unknown domain data, reduces the impact of domain offset on the model, and improves cross-scene classification performance, especially in the face of multi-source domains.
Smart Images

Figure CN118799655B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of cross-scene classification, and specifically is a domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation. Background Art
[0002] Hyperspectral images have rich spatial and spectral information, which makes them have important application value. However, in practical applications, they are affected by unpredictable real-world factors. There is often a certain domain offset between the training images and the target images, which causes the discriminator trained in the source domain to be unable to be well applied to the target domain, resulting in a decrease in the performance of the model.
[0003] Domain generalization (DG) enables the model to learn domain-invariant features when only source domain data is used for training. By reducing the inter-domain distance between single or multiple source domains, a model that is more adaptable to domain changes can be trained, thereby achieving generalization from one or more source domains to the target domain. Therefore, domain generalization can solve the problem of model performance degradation caused by inter-domain differences in the cross-scene classification task of hyperspectral images. However, existing research on the cross-scene classification task of hyperspectral images is mainly based on single-source domain generalization. It is often not enough to train the model using data from only one source domain, and effective domain generalization cannot be achieved, so domain expansion is required. At present, common single-source domain expansion is achieved by constructing simulated samples, expanding the relationship between domains, and using contrastive adversarial learning and other methods to obtain domain-invariant representations. The spatial and spectral information of hyperspectral images is decomposed, and samples are reconstructed at the visual level to achieve domain expansion. From the two aspects of data augmentation and causal alignment, a spatial-spectral augmentation and causal constraint network is proposed. By simulating the spectral deviation and spatial deviation of the target domain data, the lack of diversity of the source domain data is compensated. A causal contribution discriminator is constructed, which constructs a causal contribution vector from a causal perspective and uses contrastive learning to constrain category labels to extract "latent" causal invariance from the spectral and spatial domains.
[0004] In summary, the existing technology has fully exploited the spatial and spectral information of hyperspectral images, but most of the domain generalization research on hyperspectral images only considers the "visible" visual representation information, learns the domain-invariant representation of the spatial-spectral information level, and ignores the "invisible" frequency domain information. However, in cross-scene classification tasks, frequency domain information is less susceptible to domain shift than spatial-spectral information. Therefore, the present invention makes full use of the characteristic of frequency domain information that "is less affected by domain shift" and proposes a domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation, aiming to better learn domain-invariant representation and improve the classification performance of the model. Summary of the invention
[0005] In view of the deficiencies in the prior art, the technical problem to be solved by the present invention is to provide a domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation.
[0006] The present invention solves the technical problem by adopting the following technical solution:
[0007] A domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation includes the following steps:
[0008] Step 1: Get the original image, normalize it, and get the source domain image.
[0009] Step 2: Build a generator to generate extended domain images and intermediate domain images;
[0010] First, the source domain image is transformed from the spatial spectrum domain to the frequency domain through three-dimensional Fourier transform; the source domain image is divided into multiple image blocks with each pixel as the center; the amplitude of the image blocks is mixed, and the amplitude mixing formula is:
[0011] (3)
[0012] In the formula, Ã ( x i,sd )( u , v , m ) is the i The mixed amplitude of the image blocks, A ( x i,sd )( u , v , m ) is the i The amplitude of the image blocks before mixing, Ā ( x i,sd )( u , v , m ) is the i The mean amplitude of image blocks, ( u , v , m ) is the representation of the pixel in the frequency domain, λ is the enhancement factor;
[0013] Then, the mixed amplitude and phase of the image block are recombined to map the image block to the new frequency domain space. The new frequency domain representation of the image block is:
[0014] (4)
[0015] In the formula, F f ’ ( x i,sd )(u , v , m ) is the i The new frequency domain representation of the image patch is P ( x i,sd )( u , v , m ) is the i The phase of the image block;
[0016] Finally, the new frequency domain representation of the image block is inversely transformed into a three-dimensional Fourier transform to obtain the extended domain image;
[0017] (5)
[0018] In the formula, x ed represents the extended domain image, F f -1 represents the inverse three-dimensional Fourier transform;
[0019] The multi-source information is fused to generate the intermediate domain image. The formula is:
[0020] (6)
[0021] In the formula, x id is the intermediate domain image, w 1. w 2. w 3 are the weights of source domain, extended domain and noise domain respectively, g is Gaussian noise;
[0022] Step 3: Construct the discriminator; the discriminator consists of three sub-discriminators, each of which contains an adaptive frequency domain transformation module, a feature extractor, a classification head, and a projection head. The inputs of the three sub-discriminators are the source domain image, the extended domain image, and the intermediate domain image respectively.
[0023] For the sub-discriminator with the source domain image as input, in the adaptive frequency domain transformation module, the source domain image is added to itself after undergoing 3D Fourier transformation to obtain a mixed source domain image;
[0024] (7)
[0025] In the formula, x sd,fft represents the mixed source domain image, F f (.) represents the three-dimensional Fourier transform, Add Represents addition operation;
[0026] The mixed source domain image is passed through the feature extractor to obtain feature vectors, which are accumulated row by row to obtain a one-dimensional matrix;
[0027] (8)
[0028] In the formula, V represents a one-dimensional matrix, SUM row Indicates row-by-row accumulation operation. F e (.) represents feature extractor;
[0029] Calculate adaptive threshold based on one-dimensional matrix ε , the calculation formula is:
[0030] (9)
[0031] In the formula, V max , V min are the maximum and minimum element values of a one-dimensional matrix, respectively. δ is a floating factor;
[0032] The one-dimensional matrix is screened according to the adaptive threshold to obtain the index of the image block; the index satisfies the following formula:
[0033] (10)
[0034] In the formula, Ind { i} represents the index table, V [ i ] represents the first i elements;
[0035] According to the index table, some image blocks are selected from the source domain image and then covered with the image blocks at the corresponding positions in the source domain image after three-dimensional Fourier transformation, so as to obtain the source domain joint features containing spatial spectrum information and frequency domain information;
[0036] (11)
[0037] In the formula, x sel_fft represents the joint features of the source domain, Cover Indicates an overwrite operation. x sd ( Ind { i}) represents the image block after the source domain image is filtered, → x sd Represents the effect on the source domain image;
[0038] Similarly, the extended domain image and the intermediate domain image are respectively subjected to the corresponding adaptive frequency domain transformation modules to obtain the extended domain joint features and the intermediate domain joint features; the source domain joint features, the extended domain joint features and the intermediate domain joint features are respectively extracted by the corresponding feature extractors;
[0039] Step 4: Use source domain images, extended domain images and intermediate domain images to train the generator and discriminator. The trained adaptive frequency domain transformation module, feature extractor and classification head constitute a classification model for the classification of target domain images.
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] 1. Compared with spatial and spectral information, frequency domain information is less affected by domain shift in cross-scene classification tasks. Therefore, the present invention makes full use of the stability of frequency domain information for domain expansion, ensuring that the model can learn more robust inter-domain knowledge when facing domain shift, thereby improving the model's ability to recognize unknown domain data and solving the shortcomings of single-source domain generalization.
[0042] 2. In the cross-scene classification task of hyperspectral images, the domain offset between the extended domain and the source domain is an important factor affecting the generalization ability of the model. When generating extended samples, it is inevitable that the inter-domain gap between the extended samples and the source samples is too large, which affects the generalization ability of the model. In response to this generation randomness, existing studies have generated extended samples again through random weights to reduce the domain offset between the source domain and the extended domain. This generation randomness cannot guarantee the stability of the extended samples. Therefore, the present invention starts from the inter-domain constraints and generates intermediate domain data through self-learning. On the one hand, it eliminates the unknown caused by random weights, and at the same time ensures the reliability of the intermediate domain in narrowing the distance between the source domain and the extended domain. Considering the effectiveness of the intermediate domain, that is, whether it has unique features in the domain to improve the generalization performance of the model, Gaussian noise is added in the process of generating the intermediate domain.
[0043] 3. In the cross-scene classification task of hyperspectral images, in addition to generating reliable and effective extended data, it is also crucial to learn the information differences between multiple domains. For the learning of inter-domain differences, most existing studies consider the "visible" visual representation information in hyperspectral images, learn the domain-invariant representation of the spatial-spectral information level, and ignore the "invisible" frequency domain information. Therefore, the present invention makes full use of the advantage of frequency domain information in cross-scene tasks that it is "less affected by domain offset", decomposes the frequency domain information from the source domain image, enhances the frequency domain information, and then recombine it with the spatial spectrum information. While retaining the good spatial spectrum information of the image itself, the model can adaptively learn multiple aspects of knowledge, thereby training a more robust model. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 Schematic diagram of the model structure in the training phase;
[0045] Figure 2 The schematic diagram for generating extended domain images;
[0046] Figure 3 Schematic diagram for weight generation in the multi-source information fusion process;
[0047] Figure 4 This is the schematic diagram of the adaptive frequency domain transformation module. DETAILED DESCRIPTION
[0048] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods, but the protection scope of the present application is not limited thereto.
[0049] The present invention provides a domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation (hereinafter referred to as method, see Figures 1 - 4 ), including the following steps:
[0050] Step 1: Get the original image, normalize it, and get the source domain image.
[0051] This embodiment selects the Houston Dataset, Pavia Dataset, and Shanghai-Hangzhou Dataset as the original data. The Houston Dataset includes seven classes of unchanged scenes. 48 spectral bands are extracted from the Houston Dataset in 2018 with a wavelength range of 0.38-1.05µm, and an overlapping area of 209×955 pixels is selected as the original image. 102 spectral bands are extracted from the Pavia Dataset with a wavelength range of 430-860 nm. The image size includes 610 × 340 pixels and 1096 × 715 pixels, and the spatial resolution is 1.3 m. The Shanghai-Hangzhou Dataset contains 198 spectral bands, and the image size includes 1600×230 pixels and 590×230 pixels.
[0052] The original image is normalized by min-max normalization to map the value of each pixel to [0, 1]; then the Euclidean norm normalization is performed to ensure that the value of each pixel in each channel is relative to each other. Different pixel value ranges may have a negative impact on the convergence and stability of the model, so ensuring that the value range of all pixels is consistent through min-max normalization helps avoid this problem. Euclidean norm normalization can reduce the errors that may be caused by larger or smaller pixel values in the calculation, which helps maintain numerical stability and avoid numerical overflow or instability.
[0053] Step 2: Build a generator to generate extended domain images through intra-domain frequency domain enhancement and generate intermediate domain images through multi-source information fusion;
[0054] In order to fully maintain the integrity of spatial and spectral information in the hyperspectral image, the source domain image is transformed in the three-dimensional space by frequency domain transformation, that is, the source domain image is transformed from the spatial spectrum domain to the source domain frequency domain space by three-dimensional Fourier transform; the three-dimensional Fourier transform formula is:
[0055] (1)
[0056] In the formula, F f ( x sd )( u , v , m ) is the source domain image x sd The frequency domain representation of u , v , m ) is the representation of the pixel in the frequency domain, x sd ( h , w , c ) is the spatial spectrum domain representation of the source domain image, ( h , w , c ) is the representation of the pixel point in the spatial spectrum domain, j is an imaginary number, H , W , C are the height, width and number of channels of the source domain image respectively;
[0057] The source domain image undergoes a three-dimensional Fourier transform, and its amplitude and phase are expressed as:
[0058] (2)
[0059] In the formula, A ( x sd )( u , v , m ) represents the amplitude of the source domain image, P ( x sd )( u , v , m ) represents the phase of the source domain image, R ( x sd )( u , v, m ), I ( x sd )( u , v , m ) represent the real part and the imaginary part respectively;
[0060] Phase information has good semantic preservation characteristics, and the model that highlights phase information has better domain generalization ability. Therefore, by keeping the phase information unchanged, amplitude mixing is performed in the domain to achieve in-domain frequency domain enhancement of the source domain image. Since hyperspectral image classification is pixel-by-pixel classification, the source domain image is divided into multiple image blocks with each pixel as the center, and each image block is enhanced in the domain frequency domain to obtain the extended domain image; first, the amplitude of the image block is mixed, and the amplitude mixing formula is:
[0061] (3)
[0062] In the formula, Ã ( x i,sd )( u , v , m ) is the i The mixed amplitude of the image blocks, A ( x i,sd )( u , v , m ) is the i The amplitude of the image blocks before mixing, Ā ( x i,sd )( u , v , m ) is the i The mean amplitude of image blocks; λ is the enhancement factor, and λ ~U(0, η ), η is a hyperparameter;
[0063] Then, the mixed amplitude and phase of the image block are recombined to map the image block to the new frequency domain space. The new frequency domain representation of the image block is:
[0064] (4)
[0065] In the formula, F f ’ ( x i,sd )( u ,v , m ) is the i The new frequency domain representation of the image patch is P ( x i,sd )( u , v , m ) is the i The phase of the image block;
[0066] Finally, the new frequency domain representation of the image block is inversely transformed into a three-dimensional Fourier transform to obtain the extended domain image;
[0067] (5)
[0068] In the formula, x ed represents the extended domain image, F f -1 Represents the inverse 3D Fourier transform.
[0069] Intra-domain frequency domain enhancement is different from the inter-domain amplitude exchange operation in the prior art. The inter-domain amplitude exchange aims to exchange and reorganize the amplitudes of different source domain images in pairs, which is suitable for single-source domain generalization. However, the domain generalization hyperspectral image classification task is aimed at multi-source domain generalization, so the inter-domain amplitude exchange is not applicable. The intra-domain frequency domain enhancement makes the generated extended domain image more suitable for the domain generalization hyperspectral image classification task of the present invention.
[0070] The domain generalization task aims to learn to solve the domain shift problem, that is, to train the model's generalization ability for unknown domains from certain domain shift data, and the model robustness depends on domain shift. This phenomenon is particularly obvious in the domain generalization hyperspectral image classification task. The domain shift between the extended domain and the source domain is an important factor affecting the generalization ability of the model. Therefore, multi-source information is fused, and the intermediate domain is generated by autonomously learning the source domain and extended domain features, and then the distance between the two domains is shortened when the source domain and the extended domain have too large domain shifts, thereby realizing inter-domain constraints. At the same time, Gaussian noise is added in the process of generating the intermediate domain. On the premise of ensuring the effectiveness of the intermediate domain, it also has certain unique characteristics, thereby improving the generalization ability of the model for unknown domains. The intermediate domain image generation formula is:
[0071] (6)
[0072] In the formula, x id is the intermediate domain image, w 1. w 2. w 3 are the weights of the source domain, extended domain and noise domain, and their value ranges are all (0,1); g is Gaussian noise, , μ , σ denote mean and variance respectively;
[0073] The weight generation process of the three domains is as follows: the source domain image, the extended domain image and the Gaussian noise are added together, and the added image is sequentially subjected to 3×3 convolution, Relu activation function and center pooling, and then to three fully connected layers and Sigmoid activation function respectively to obtain the weights of the source domain, extended domain and noise domain. w 1. w 2 and w 3. Center pooling can better obtain key information for pixel-level classification tasks such as hyperspectral images.
[0074] Step 3: Construct the discriminator; the discriminator contains three sub-discriminators, each of which contains an adaptive frequency domain transformation module, a feature extractor, a classification head, and a projection head. The adaptive frequency domain transformation module and the feature extractor between the sub-discriminators share weights respectively, and the source domain image, the extended domain image, and the intermediate domain image are used as the inputs of the three sub-discriminators respectively. The adaptive frequency domain transformation module aims to autonomously learn through threshold screening to obtain joint features containing spatial spectrum information and frequency domain information, and realize the coordination between the spatial spectrum domain and the frequency domain. The feature extractor is used for further feature extraction, which is composed of two feature extraction blocks stacked, each of which includes a convolution layer, a Relu activation function, and a maximum pooling layer. The classification head is used for classification tasks, and the projection head is used to train contrast and adversarial losses to learn domain-invariant representations of categories.
[0075] Taking the sub-discriminator with the source domain image as input as an example, in the adaptive frequency domain transformation module, the source domain image is added to itself after undergoing a three-dimensional Fourier transform to obtain a mixed source domain image;
[0076] (7)
[0077] In the formula, x sd,fft represents the mixed source domain image, F f (.) represents the three-dimensional Fourier transform, Add Represents addition operation;
[0078] The mixed source domain image is passed through the feature extractor to obtain a feature vector, which is accumulated row by row to obtain a one-dimensional matrix with a dimension of b. The one-dimensional matrix is regarded as the quantization of the input b image blocks;
[0079] (8)
[0080] In the formula, V represents a one-dimensional matrix, SUM rowIndicates row-by-row accumulation operation. F e (.) represents feature extractor;
[0081] Calculate adaptive threshold based on one-dimensional matrix ε , the calculation formula is:
[0082] (9)
[0083] In the formula, V max , V min They are the maximum and minimum element values of a one-dimensional matrix respectively; δ It is a floating factor used to control the floating degree of the adaptive threshold. Its value is obtained through self-learning. It ensures the accuracy of the image while determining the screening range. image The information interaction of the blocks, that is, the one-dimensional matrix passes through three fully connected layers in sequence to obtain a 1×1 feature vector, which can be used to obtain a floating factor after the Sigmoid activation function. δ .
[0084] Based on adaptive threshold ε For a one-dimensional matrix V Filter and obtain the index of the image block; the index satisfies the following formula:
[0085] (10)
[0086] In the formula, Ind { i} represents the index table, V [ i ] represents the first i elements, i The value range of is 0 to b;
[0087] According to the index table, some image blocks are selected from the source domain image and then covered with the image blocks at the corresponding positions in the source domain image after three-dimensional Fourier transformation, so as to obtain the source domain joint features containing spatial spectrum information and frequency domain information;
[0088] (11)
[0089] In the formula, x sel_fft represents the joint features of the source domain, Cover Indicates an overwrite operation. x sd ( Ind { i}) represents the image block after the source domain image is filtered, → x sd Represents the effect on the source domain image.
[0090] Similarly, the extended domain image and the intermediate domain image are respectively passed through the corresponding adaptive frequency domain transformation modules to obtain the extended domain joint features and the intermediate domain joint features; the source domain joint features, the extended domain joint features and the intermediate domain joint features are respectively passed through the corresponding feature extractors for further feature extraction, and the extracted features are classified through the classification head.
[0091] Step 4: Use the source domain images, extended domain images, and intermediate domain images to train the generator and discriminator, and optimize the generator and discriminator. The optimization of the generator and discriminator is achieved through classification loss and contrast and adversarial loss. The classification loss uses cross entropy loss, and the calculation formula is:
[0092] (12)
[0093] In the formula, L ce ( p k , y k ) represents the cross entropy loss, p k Representation sample x k The classification probability, y k Representation sample x k Tags, y k c Representation sample x k About Categories c Tags, p k c Representation sample x k Belongs to category c probability;
[0094] Based on cross entropy loss, classification loss L com It is expressed as:
[0095] (13)
[0096] In the formula, N Indicates the sample size.
[0097] In order to prevent excessive separation and aggregation within and between domains, supervised contrastive adversarial learning is introduced to encourage the discriminator to learn domain-invariant representations. Therefore, the contrastive loss and adversarial loss obtained by the projection head are expressed as:
[0098] (14)
[0099] In the formula, L con Represents contrast loss, which makes the distance between samples of the same type closer and the distance between samples of different types farther, that is, the model learns the shared representation of samples of the same type; L adv Represents adversarial loss, ensuring that the extended domain has specific domain information; P ( k ) represents the set of all positive samples, A ( k ) represents the set of all negative samples, Z k Represents the original feature information of the sample, Z p + Indicates p Positive samples, S represents matrix multiplication, τ is a constant, Z a - Indicates a negative samples, n c Indicates that it belongs to the category c The sample size, P c ( k ) indicates belonging to the category c The positive sample set, A c ( k ) indicates belonging to the category c A set of negative samples;
[0100] The optimization process of the discriminator includes the combined optimization of the feature extractor, classification head and projection head, so the optimization function of the discriminator is:
[0101] (15)
[0102] In the formula, θ f , θ c , θ p denote the parameters of the feature extractor, classification head, and projection head, respectively. L com,SD represents the source domain classification loss, L com,ED represents the extended domain classification loss, L com,ID represents the intermediate domain classification loss, λ 1 is a hyperparameter used to balance the contrastive loss.
[0103] The optimized function of the generator is:
[0104] (16)
[0105] In the formula, θ G represents the parameters of the generator, λ 2 is a hyperparameter used to balance the adversarial loss.
[0106] The trained adaptive frequency domain transform module, feature extractor and classification head constitute a classification model. The target domain image is input into the classification model, and the classification result is output to achieve the classification of domain generalized hyperspectral images.
[0107] In order to verify the effectiveness of the proposed method, the method was tested on three datasets: Houston, Pavia and Shanghai-Hangzhou. At the same time, some domain adaptive classification models (MRAN [1] 、DSAN [2] 、TsTnet [6] ) and the domain generalization classification model (PDEN [3] ,LDSDG [4] 、SagNet [5] 、SDEnet [7] ,LLURnet [8] 、S2ECnet [9] ), and the model performance is measured by three indicators: overall classification accuracy (OA), average classification accuracy (AA), and Kappa coefficient. Test experiment 1 uses the Houston2013 dataset as the source domain and the Houston2018 dataset as the target domain. The classification results are shown in Table 1. Test experiment 2 uses the PaviaU dataset as the source domain and the PaviaC dataset as the target domain. The classification results are shown in Table 2. Test experiment 3 uses the Hangzhou dataset as the source domain and the Shanghai dataset as the target domain. The classification results are shown in Table 3.
[0108]
[0109]
[0110]
[0111] The above test results show that the classification model (TFTnet) of the present invention has achieved the best classification performance on all data sets. Compared with the other models, the three evaluation indicators have been improved to varying degrees. It is worth noting that compared with the mainstream field generalization models SDEnet, LLURnet and S2 For ECnet, OA increased by 0.42% to 4.05%, AA increased by 0.14% to 1.96%, and Kappa coefficient increased by 0.59% to 4.73%. This is because the present invention incorporates the learning of frequency domain information and exhibits stronger stability when facing the domain shift problem. Therefore, the present invention solves the domain shift problem well and improves the classification performance.
[0112] The sources of each model are:
[0113] [1] Y. Zhu et al., “Multi-representation adaptation network for cross-domain image classification,” Neural Netw., vol. 119, pp. 214–221, Nov.2019.
[0114] [2] Y. Zhu et al., "Deep subdomain adaptation network for image classification," IEEE Trans. Neural Netw. Learn. Syst., vol. 32, no. 4, pp.1713–1722, Apr. 2021.
[0115] [3] L. Li et al., “Progressive domain expansion network for singledomain generalization,” in Proc. IEEE / CVF Conf. Comput. Vis. PatternRecognit. (CVPR), Jun. 2021, pp. 224–233.
[0116] [4] Z. Wang, Y. Luo, R. Qiu, Z. Huang, and M. Baktashmotlagh, “Learning to diversify for single domain generalization,” in Proc. IEEE / CVFInt. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 814–823.
[0117] [5] H. Nam, H. Lee, J. Park, W. Yoon, and D. Yoo, “Reducing domaingap by reducing style bias,” in Proc. IEEE / CVF Conf. Comput. Vis. PatternRecognit. (CVPR), Jun. 2021, pp. 8686–8695.
[0118] [6] Y. Zhang, W. Li, M. Zhang, Y. Qu, R. Tao and H. Qi, "TopologicalStructure and Semantic Information Transfer Network for Cross-SceneHyperspectral Image Classification," in IEEE Transactions on Neural Networksand Learning Systems, vol. 34, no. 6, pp. 2817-2830, June 2023.
[0119] [7] Y. Zhang, W. Li, W. Sun, R. Tao, and Q. Du, “Single-source domainexpansion network for cross-scene hyperspectral image classification,” IEEETrans. Image Process., vol. 32, pp. 1498–1512, 2023.
[0120] [8] H. Zhao, J. Zhang, L. Lin, J. Wang, S. Gao, and Z. Zhang,“Locally linear unbiased randomization network for cross-scene hyperspectralimage classification,” IEEE Trans. Geosci. Remote Sens., vol. 61, 2023, Art.no. 5526512.
[0121] [9] L. Dong, J. Geng and W. Jiang, "Spectral–Spatial Enhancement andCausal Constraint for Hyperspectral Image Cross-Scene Classification," inIEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1-13, 2024,Art no. 5507013, doi: 10.1109 / TGRS.2024.3360103.
[0122] Any matters not described in the present invention are applicable to the prior art.
Claims
1. A domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation, characterized in that: The method comprises the following steps: Step 1: Get the original image, normalize it, and get the source domain image. Step 2: Build a generator to generate extended domain images and intermediate domain images; First, the source domain image is transformed from the spatial spectrum domain to the frequency domain through three-dimensional Fourier transform; the source domain image is divided into multiple image blocks with each pixel as the center; the amplitude of the image blocks is mixed, and the amplitude mixing formula is: (3) In the formula, Ã ( x i,sd )( u , v , m ) is the i The mixed amplitude of the image blocks, A ( x i,sd )( u , v , m ) is the i The amplitude of the image blocks before mixing, Ā ( x i,sd )( u , v , m ) is the i The mean amplitude of image blocks, ( u , v , m ) is the representation of the pixel in the frequency domain, λ is the enhancing factor; Then, the mixed amplitude and phase of the image block are recombined to map the image block to the new frequency domain space. The new frequency domain representation of the image block is: (4) In the formula, F f ’ ( x i,sd )( u , v , m ) is the i The new frequency domain representation of the image patch is P ( x i,sd )( u , v , m ) is the i The phase of the image block; Finally, the new frequency domain representation of the image block is inversely transformed into a three-dimensional Fourier transform to obtain the extended domain image; (5) In the formula, x ed represents the extended domain image, F f -1 represents the inverse three-dimensional Fourier transform; The multi-source information is fused to generate the intermediate domain image. The formula is: (6) In the formula, x id is the intermediate domain image, x sd is the source domain image, w 1. w 2. w 3 are the weights of source domain, extended domain and noise domain respectively, g is Gaussian noise; Step 3: Build a discriminator; the discriminator contains three sub-discriminators, and the inputs of the three sub-discriminators are the source domain image, the extended domain image, and the intermediate domain image respectively; Step 4: Use source domain images, extended domain images and intermediate domain images to train the generator and discriminator. The trained adaptive frequency domain transformation module, feature extractor and classification head constitute a classification model for the classification of target domain images.
2. The domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation according to claim 1 is characterized in that: The sub-discriminator comprises an adaptive frequency domain transformation module, a feature extractor, a classification head and a projection head; for the sub-discriminator taking the source domain image as input, in the adaptive frequency domain transformation module, the source domain image is subjected to a three-dimensional Fourier transformation and then added to itself to obtain a mixed source domain image; (7) In the formula, x sd,fft represents the mixed source domain image, F f (.) represents the three-dimensional Fourier transform, Add Represents addition operation; The mixed source domain image is passed through the feature extractor to obtain feature vectors, which are accumulated row by row to obtain a one-dimensional matrix; (8) In the formula, V represents a one-dimensional matrix, SUM row Indicates row-by-row accumulation operation. F e (.) represents feature extractor; Calculate adaptive threshold based on one-dimensional matrix ε , the calculation formula is: (9) In the formula, V max , V min are the maximum and minimum element values of a one-dimensional matrix, respectively. δ is a floating factor; The one-dimensional matrix is screened according to the adaptive threshold to obtain the index of the image block; the index satisfies the following formula: (10) In the formula, Ind { i } represents the index table, V [ i ] represents the first i elements; According to the index table, some image blocks are selected from the source domain image and then covered with the image blocks at the corresponding positions in the source domain image after three-dimensional Fourier transformation, so as to obtain the source domain joint features containing spatial spectrum information and frequency domain information; (11) In the formula, x sel_fft represents the joint features of the source domain, Cover Indicates an overwrite operation. x sd ( Ind { i }) represents the image block after the source domain image is filtered, → x sd Represents the effect on the source domain image; Similarly, the extended domain image and the intermediate domain image are respectively subjected to the corresponding adaptive frequency domain transformation modules to obtain the extended domain joint features and the intermediate domain joint features; the source domain joint features, the extended domain joint features and the intermediate domain joint features are respectively extracted through the corresponding feature extractors.
3. The domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation according to claim 1 or 2, characterized in that: The optimization function of the discriminator is: (15) In the formula, θ f , θ c , θ p denote the parameters of the feature extractor, classification head, and projection head, respectively. L com,SD represents the source domain classification loss, L com,ED represents the extended domain classification loss, L com,ID represents the intermediate domain classification loss, L con represents the contrast loss, λ 1 is a hyperparameter; The optimized function of the generator is: (16) In the formula, θ G represents the parameters of the generator, L adv Represents resistance to loss, λ 2 is a hyperparameter.
4. The domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation according to claim 3 is characterized in that: The generation process of the source domain, extended domain and noise domain weights is as follows: the source domain image, the extended domain image and Gaussian noise are added, and the added image is sequentially subjected to 3×3 convolution, Relu activation function and center pooling, and then respectively subjected to three fully connected layers and Sigmoid activation function to obtain the source domain, extended domain and noise domain weights.
5. The domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation according to claim 2 is characterized in that: The generation process of the floating factor is as follows: a one-dimensional matrix passes through three fully connected layers in sequence to obtain a 1×1 feature vector, and the feature vector is passed through a Sigmoid activation function to obtain a floating factor.
6. The domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation according to claim 1 or 5, characterized in that: The feature extractor is composed of two stacked feature extraction blocks, each of which includes a convolutional layer, a Relu activation function and a maximum pooling layer.
7. The domain generalization hyperspectral image classification method based on three-dimensional frequency domain transformation according to claim 1 is characterized in that: The normalization process includes two types: minimum-maximum normalization and Euclidean norm normalization.
Citation Information
Patent Citations
Living body detection method, device and equipment
CN116486493A
Medical image processing system based on spectrum enhancement
CN117876307A
Cross-domain generalization image recognition method based on structure and style augmentation
CN118038150A
Detecting forged facial images using frequency domain information and local correlation
US20230081645A1
Cited By
Hyperspectral image generalization classification method and device based on dynamic frequency domain mixing and progressive decoupling
CN121861492A
