A hyperspectral image sparse small target data enhancement and background fusion method

Abstract

The application discloses a hyperspectral image sparse small target data enhancement and background fusion method. The method first pre-processes and performs principal component analysis dimension reduction on the hyperspectral data, and divides the hyperspectral data into multiple groups of three-channel pseudo-color images to adapt to the input of a two-dimensional generative adversarial network. Then, the target and background images are fused in the pseudo-color domain, and are input into an improved StyleGAN2-ADA model together with target condition parameters for training, so that controllable generation of small target attributes (category, quantity and position) is realized. After training, the model synthesizes multiple groups of pseudo-color images, and finally reconstructs a hyperspectral data cube with complete wavebands through mapping and principal component analysis inverse transformation. The application solves the problems of poor compatibility between high-dimensional data and a generative model, weak controllability of small target generation and unnatural fusion, can generate high-quality data with spectral fidelity and spatial consistency, and effectively alleviates the problem of sample scarcity in hyperspectral sparse small target detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for enhancing sparse small target data and fusing background in hyperspectral images. Background Technology

[0002] Hyperspectral imaging technology, with its "image-spectrum integration" characteristic, has irreplaceable value in sparse small target detection tasks in fields such as military reconnaissance, environmental monitoring, and precision agriculture. However, acquiring hyperspectral target samples, especially sparse small target samples, is costly, time-consuming, and difficult to annotate, resulting in an extreme scarcity of high-quality annotated datasets. This severely restricts the performance improvement and generalization ability expansion of deep learning-based hyperspectral target detection algorithms.

[0003] To address the problem of data scarcity, Generative Adversarial Networks (GANs) have become one of the core technologies for hyperspectral data augmentation. Through adversarial training between the generator and discriminator, GANs can learn the distribution of real data and generate synthetic samples. Currently, related technologies mainly revolve around three directions: "adapting general GAN ​​models to hyperspectral data," "generating small sample data," and "fusion of target and background," forming two core technology branches:

[0004] Hyperspectral data generation technologies: These technologies focus on the spectral fidelity and spatial structure consistency of hyperspectral images. They adapt the three-dimensional cubic (x,y,λ) characteristics of hyperspectral data by improving the GAN network architecture. Representative solutions include spatial-spectral co-generation methods based on Cascade conditional GANs and direct generation technologies of hyperspectral data based on 3D-GANs.

[0005] Few-shot generation techniques: To address the overfitting problem during training with small datasets, adaptive enhancement or regularization mechanisms are developed. Among them, the StyleGAN2-ADA model, with its adaptive discriminator enhancement (ADA) mechanism, has performed outstandingly in few-shot generation tasks in fields such as faces and medical images. It can stabilize the training process by dynamically adjusting the data enhancement intensity and generate high-quality, highly diverse synthetic samples.

[0006] Despite the progress made by existing technologies in their respective fields, there are still significant limitations in the task of "generating hyperspectral data for embedding sparse small targets into complex backgrounds in small-sample scenarios": most hyperspectral generation technologies do not consider the weak feature learning and controllable embedding requirements of sparse small targets, while small-sample generation technologies (such as StyleGAN2-ADA) are mainly aimed at RGB or general 2D images and are not adapted to the core characteristics of hyperspectral data such as high-dimensional interspectral correlation and spatial-spectral consistency. A complete technical solution that can take into account "small-sample adaptability", "spectral fidelity" and "target embedding authenticity" has not yet been formed.

[0007] Detailed description of the closest existing technical solution

[0008] Option 1: A general few-shot image generation technique based on StyleGAN2-ADA

[0009] Technical source: StyleGAN2-ADA model (related research was published in "Training Generative Adversarial Networks with Limited Data", and through open source code, it has been extended to small sample generation tasks in fields such as medical images and biometric images (such as "Enhancing automated strabismus classification with limited data: Data augmentation using StyleGAN2-ADA" and "Abdominal MRI Synthesis using StyleGAN2-ADA").

[0010] Technical Principle: This technology is an improved version of the StyleGAN2 model. Its core innovation lies in introducing an Adaptive Discriminator Enhancement (ADA) mechanism to address the discriminator overfitting problem during small-sample training. Its core architecture includes:

[0011] Mapping Network: Maps random noise vectors from the Z-space to the decoupled W-space latent space, enabling fine-grained control over the semantic attributes of the generated images. Style-based Generator: Injects the W-space latent code into each layer of the network through the Adaptive Instance Normalization (AdaIN) operation, controlling image styles (texture, color, lighting, etc.) at different scales. ADA Mechanism: Constructs a differentiable data augmentation pipeline, including pixel-level augmentation (horizontal flip, integer multiple rotation), geometric transformation (scaling, cropping), color transformation (brightness, contrast adjustment), etc., and dynamically adjusts the augmentation probability based on overfitting metrics during training (such as FID score divergence), forcing the discriminator to learn more robust high-level features and avoiding overfitting.

[0012] Implementation process:

[0013] Data preparation: Collect a small number (thousands or even hundreds) of real image samples, divide them into training set and validation set, and preprocess them into 2D images of uniform size (such as RGB three-channel format).

[0014] Model training: Based on the open-source architecture of StyleGAN2-ADA, non-saturated adversarial loss and path length regularization loss are configured, and adversarial training is performed using the Adam optimizer. During the training process, the ADA mechanism automatically monitors the overfitting state and adjusts the enhancement intensity.

[0015] Sample generation: After training converges, a synthetic image that conforms to the distribution of real data is generated by inputting a random noise vector into the mapping network, which is used to supplement the training dataset.

[0016] Technical limitations:

[0017] Insufficient data adaptability: The original model was designed for 3-channel RGB images and did not take into account the high-dimensional band characteristics of hyperspectral data (tens to hundreds of channels), so it could not directly process three-dimensional hyperspectral data cubes;

[0018] The enhancement operation is limited: it only supports color and geometric transformations of traditional RGB images, and does not design a physically reasonable spectral enhancement strategy for the interspectral correlation of hyperspectral data, resulting in generated samples lacking spectral authenticity.

[0019] Missing target control: It belongs to the unconditional generation model or simple category conditional generation model, which cannot achieve fine control over the number, position and pose of sparse small targets, nor does it consider the seamless integration of targets and background, making it difficult to generate hyperspectral scene samples with "sparse small targets embedded in complex background".

[0020] Option 2: Hyperspectral data generation technology based on Cascade conditional GAN

[0021] Technical source: The relevant research was published in "Cascade conditional generative adversarialnets for spatial-spectral hyperspectral sample generation", which is a representative conditional generation scheme in the field of hyperspectral data generation.

[0022] Technical Principle: This scheme targets the spatial-spectral synergistic characteristics of hyperspectral data, designing a cascaded conditional GAN ​​architecture. A multi-stage generator progressively optimizes the spectral curves and spatial structure, while the discriminator simultaneously evaluates spectral fidelity and spatial consistency to ensure the physical plausibility of the generated samples. Its core design includes: a cascaded generator: divided into a spectral generation stage and a spatial optimization stage. The first stage generates initial spectral data that conforms to the spectral characteristics of real ground objects, and the second stage optimizes spatial texture details based on the spectral data; a conditional constraint mechanism: using ground object category labels as conditions, guiding the generator to generate hyperspectral images of specific categories to ensure semantic consistency; and a discriminator design: employing a dual-branch discriminant structure to separately judge the correlation in the spectral dimension and the structural similarity in the spatial dimension, enhancing the spectral synergy of the generated samples.

[0023] Implementation process:

[0024] Data preparation: Collect real hyperspectral datasets (such as AVIRIS and PaviaUniversity datasets), extract land cover category labels, and perform radiometric calibration, atmospheric correction, and normalization preprocessing on the data;

[0025] Model training: Construct a Cascade conditional GAN ​​network, configure adversarial loss and spectral angle matching (SAM) loss, train it with land cover category as the condition, and optimize the generator and discriminator parameters through multi-stage iterative optimization;

[0026] Sample generation: Input the target land cover category label and generate hyperspectral image samples of the corresponding category to expand the training dataset for hyperspectral classification or detection tasks.

[0027] Technical limitations:

[0028] Poor adaptability to small datasets: No anti-overfitting mechanism was designed for small datasets. When the number of training samples is insufficient (e.g., less than 1,000 images), pattern collapse or insufficient diversity of generated samples may occur.

[0029] Lack of handling for sparse small targets: Focusing on the generation of conventional land cover categories, the generator does not consider the characteristics of sparse small targets, such as "weak features and low sample proportion". The generator tends to learn background features and ignore target features.

[0030] Insufficient embedding flexibility: It can only generate complete images of a single land cover category, and cannot achieve "controllable fusion of sparse small targets and complex backgrounds", making it difficult to simulate the complex situation of sparse small targets randomly distributed and interacting with the background in real scenes.

[0031] Option 3: Hyperspectral image data generation technology based on semi-supervised WGAN-GP

[0032] Technology Source: This technology originates from patent CN109389080A, which discloses a method based on semi-supervised WGAN-GP (Generating Adversarial Network-GP) for hyperspectral image data enhancement, particularly for hyperspectral data augmentation in situations of scarce samples. This method effectively improves the performance of hyperspectral image classification and alleviates the limited sample problem by generating hyperspectral samples through adversarial training. This technology has been applied in various fields, including environmental monitoring and precision agricultural management, especially in target detection and classification tasks within the field of hyperspectral image analysis.

[0033] Technical Principle: This technology employs the WGAN-GP framework, which combines Generative Adversarial Networks (WGAN) with Gradient Penalty (GP). This method utilizes semi-supervised learning, combining labeled and unlabeled hyperspectral image data to generate synthetic hyperspectral data through a generative adversarial network, thereby enhancing the generalization ability of the classifier. The core architecture includes:

[0034] The generator: It generates pseudo-hyperspectral images that match the distribution of real data, using raw samples and noise input from the hyperspectral image. The generator is constructed using a multi-layer convolutional neural network (CNN) and is able to capture the spatial and spectral features of the hyperspectral image.

[0035] Discriminator: Employing an improved WGAN-GP structure, the discriminator not only distinguishes between real and fake samples but also incorporates a gradient penalty term. Regularization is used to improve training stability, prevent mode collapse, and enhance the evaluation capability of generated images.

[0036] Semi-supervised learning mechanism: By combining labeled and unlabeled data for joint training, it learns to generate more diverse synthetic samples using features from the unlabeled data. The discriminator classifies real and pseudo-generated data during training and provides feedback to the generator, improving the quality of generated images.

[0037] Implementation process:

[0038] 1. Data Preparation: First, collect a small number (hundreds to thousands) of real hyperspectral image samples. Perform preprocessing on these data, including radiometric calibration and atmospheric correction, to ensure data quality.

[0039] 2. Model Training: A semi-supervised WGAN-GP framework is adopted, configuring non-saturating adversarial loss and gradient penalty loss, and using the Adam optimizer for adversarial training. During training, the gradient penalty term is used to prevent the discriminator from overfitting, thereby improving the robustness of the generator and image quality.

[0040] 3. Sample Generation: After training, the trained generator is used to input random noise to generate synthetic images that conform to the characteristics of the hyperspectral data. These images can be used to supplement the training set and help improve the performance of the hyperspectral image classifier.

[0041] Technical limitations:

[0042] Insufficient data adaptability: Although this method can generate hyperspectral images, the generated hyperspectral data may have some deviations in terms of interspectral correlation and spectral fidelity because it is not specifically optimized for the high-dimensional band characteristics of hyperspectral images. In particular, when generating hyperspectral images containing sparse small targets, it may not be able to fully match the complex background in the real scene.

[0043] Limitations of the enhancement operations: This method only supports geometric transformations and color adjustments of hyperspectral images, and does not design inter-spectral enhancement strategies tailored to the characteristics of hyperspectral images. Therefore, the generation process may result in a lack of realism in the spectral information.

[0044] Insufficient Target-Background Fusion: Since this method mainly focuses on the image generation process, it fails to fully consider the seamless fusion of the target and background in the generated hyperspectral image. It is difficult to achieve the number of sparse small targets, precise position control and attitude adjustment. The generated hyperspectral image may have unclear target features or excessive background information interference. Summary of the Invention

[0045] The technical solution of this invention to solve the above-mentioned technical problems is to provide a method for enhancing sparse small target data and fusing background in hyperspectral images, comprising the following steps:

[0046] Step S1: Hyperspectral data preprocessing. Logarithmic transformation and Laplacian transformation are performed on the input hyperspectral data containing the target and background to enhance details, and image scaling is performed to unify spatial size, resulting in preprocessed hyperspectral segmentation data.

[0047] Step S2: Dimensionality reduction is performed on the preprocessed hyperspectral segmentation data, the high-dimensional band data is compressed into a K-channel principal component feature map, and the K-channel principal component feature map is divided into M groups of three-channel pseudo-color images;

[0048] Step S3: Fuse the pseudo-color image representing the target with the pseudo-color image representing the background in the pseudo-color domain to generate multiple realistic fused pseudo-color training images;

[0049] Step S4: Input the real fused pseudo-color training image and its corresponding conditional control parameters into the improved StyleGAN2-ADA model for adversarial training. The conditional control parameters include at least target category, quantity, and location information. After training, the improved StyleGAN2-ADA model generates a group of synthetic pseudo-color images containing sparse small targets.

[0050] Step S5: Map the synthesized pseudo-color image group back to a K-channel synthesized principal component feature map, and perform inverse principal component analysis on the K-channel synthesized principal component feature map to regenerate an N-band synthesized hyperspectral data cube.

[0051] Step S6: Perform target detection verification on the synthesized hyperspectral data cube and output the detection results.

[0052] Further, in step S1, the formula for the logarithmic transformation is:

[0053]

[0054] in, =1e-8, For the normalized data, The transformed data;

[0055] The Laplace transform uses a 3×3 Laplace kernel logarithmic transform to perform convolution operations on the data to enhance the edges.

[0056] Furthermore, in step S2, principal component analysis is used for dimensionality reduction, and the dimensionality reduction formula is as follows:

[0057]

[0058] Where E is the PCA transformation matrix, It is the mean spectrum; the PCA model parameters 𝐸 and mean spectrum 𝜇 are obtained by jointly fitting the preprocessed hyperspectral data of the target and background samples in the training set;

[0059] The dimensionality-reduced data is arranged according to spatial dimensions to obtain K-channel principal component feature maps, and pseudo-color images are formed by grouping every three channels. When K is not an integer multiple of 3, the last group is supplemented to three channels by copying or padding.

[0060] Furthermore, in step S3, the fusion employs Alpha mixing, with the following formula:

[0061]

[0062] Where α=0.9, and the transparency factor α∈[0.85,0.95], we obtain 𝑀 true-to-life pseudo-color images. .

[0063] Further, in step S4, the generator of the improved StyleGAN2-ADA model includes a mapping network and a synthesis network. The conditional control parameters and random noise are input into the mapping network to generate a conditional style vector. The synthesis network includes a self-attention module for capturing the global correlation between the target and the background.

[0064] Furthermore, in step S4, the loss functions used in adversarial training include non-saturated adversarial loss, path length regularization loss, and RGB channel consistency loss, wherein the RGB channel consistency loss is constructed based on the Pearson correlation coefficient between each channel of the generated image and the real image.

[0065] Further, in step S5, the three channels of each group forming the pseudo-color image are restored to their corresponding partial K-channel data through linear mapping, and the complete K-channel synthetic principal component feature map is obtained after splicing all groups of data; the formula for the inverse PCA transform is:

[0066]

[0067] in, It is the recovered original spectrum. Synthesis is the K-channel principal component features of each pixel, μ is the mean spectrum during PCA training, and 𝐸 is the transformation matrix of PCA, which transforms K-dimensional data into N-dimensional data.

[0068] Furthermore, step S6 specifically includes:

[0069] S61: The synthesized hyperspectral data cube is processed by the spatial attention module to generate an attention weight map highlighting the target region, and the weight map is multiplied with the original data to obtain the enhanced data;

[0070] S62: Input the enhanced data into the three-dimensional convolutional neural network detection model to extract spatial-spectral joint features and predict the target bounding box and category;

[0071] S63: Perform non-maximum suppression on all predicted target bounding boxes, filter out overlapping boxes, and output the final target detection result.

[0072] Compared with the prior art, the technical solution of the present invention has the following technical effects:

[0073] (1) The compatibility problem between high-dimensional spectral data and the 2DGAN framework is solved by using the "band dimensionality reduction + channelization" strategy;

[0074] This invention employs Principal Component Analysis (PCA) or Minimum Noise Separation (MNF) for band dimensionality reduction during the data preprocessing stage, retaining core bands with a cumulative contribution rate ≥95% (e.g., 64 bands). This removes redundant information, reduces computational burden, and maintains high inter-spectral correlation. Simultaneously, the N dimensionality-reduced bands are converted into an N-channel tensor format, adapting the hyperspectral data cube (H×W×λ) to the input requirements of 2D convolutional networks. This design directly addresses the model training collapse problem caused by the curse of dimensionality and, through information preservation strategies during dimensionality reduction, ensures that the spectral curves of the generated samples are smooth and continuous, conforming to physical laws.

[0075] (2) By using "dual condition injection + feature balancing design", the problem of imbalance between fine control and feature learning of sparse small targets is solved;

[0076] This invention transforms category labels into 512-dimensional features through embedding vectors. These features are then concatenated with random noise vectors and injected into a mapping network. Furthermore, affine parameters generated by a small MLP are injected into each AdaIN layer of the synthesis network, achieving basic control over the presence or absence of targets. Advanced schemes can also incorporate conditions such as Gaussian heatmaps and pose angles to achieve fine-grained control over the number, location, and pose of targets. Simultaneously, by constructing a target sample library and a background sample library (with a ratio of 1:3-5), and strengthening the modulation weight of target features during conditional generation, the generator avoids ignoring the weak features of sparse small targets due to an excessively high proportion of background pixels. This balances the learning priorities of background and target, solving the problems of low accuracy and poor recognition in the generation of sparse small targets.

[0077] (3) By using the “generation-extraction-learning fusion” process, the problem of seamless fusion of target and background and spatial-spectral consistency is solved;

[0078] This invention first uses a segmentation network to accurately extract the spectral and shape information of the generated target. Then, it utilizes a small U-Net fusion network, taking "pure background + target spectral patch + location mask" as input, and trains it using L1 loss and spectral angle matching (SAM) loss to achieve a natural fusion of the target and background, generating realistic shadows and edge transitions. This design avoids the abrupt edge problems caused by simple pasting. At the same time, the training process of the fusion network forcibly constrains the synergy between spatial texture and spectral features, ensuring the spatial continuity of the spectral curves of objects in the same area, maintaining the spatial-spectral consistency of the generated scene, and meeting the requirements of downstream detection tasks for sample authenticity. Attached Figure Description

[0079] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0080] Figure 1 This is a flowchart illustrating the overall technical process of the hyperspectral image sparse small target data enhancement and background fusion method described in this invention.

[0081] Figure 2 This is a schematic diagram of the pseudo-color RGB segmentation of the present invention;

[0082] Figure 3 This is a comparison chart of the spectral reconstruction effects of the present invention;

[0083] Figure 4 This is a diagram of the StyleGAN2-ADA network structure of the present invention. Detailed Implementation

[0084] This invention proposes a method for enhancing sparse small target data and fusing background in hyperspectral images. It aims to propose a hyperspectral data dimensionality adaptation strategy of band dimensionality reduction and channelization to solve the compatibility problem between high-dimensional data and the 2D GAN framework, while preserving interspectral correlation.

[0085] The method for enhancing sparse small target data and fusing background in hyperspectral images proposed in this invention will be described below in specific embodiments:

[0086] Example 1:

[0087] In the technical solution of this embodiment, such as Figure 1 As shown, a method for enhancing sparse small target data and fusing background in hyperspectral images includes the following steps:

[0088] Step S1: Hyperspectral data preprocessing. Logarithmic transformation and Laplacian transformation are performed on the input hyperspectral data containing the target and background to enhance details, and image scaling is performed to unify spatial size, resulting in preprocessed hyperspectral segmentation data.

[0089] Step S2: Dimensionality reduction is performed on the preprocessed hyperspectral segmentation data, the high-dimensional band data is compressed into a K-channel principal component feature map, and the K-channel principal component feature map is divided into M groups of three-channel pseudo-color images;

[0090] Step S3: Fuse the pseudo-color image representing the target with the pseudo-color image representing the background in the pseudo-color domain to generate multiple realistic fused pseudo-color training images;

[0091] Step S4: Input the real fused pseudo-color training image and its corresponding conditional control parameters into the improved StyleGAN2-ADA model for adversarial training. The conditional control parameters include at least target category, quantity, and location information. After training, the improved StyleGAN2-ADA model generates a group of synthetic pseudo-color images containing sparse small targets.

[0092] Step S5: Map the synthesized pseudo-color image group back to a K-channel synthesized principal component feature map, and perform inverse principal component analysis on the K-channel synthesized principal component feature map to regenerate an N-band synthesized hyperspectral data cube.

[0093] Step S6: Perform target detection verification on the synthesized hyperspectral data cube and output the detection results.

[0094] Further, in step S1, the formula for the logarithmic transformation is:

[0095]

[0096] in, =1e-8, For the normalized data, The transformed data;

[0097] The Laplace transform uses a 3×3 Laplace kernel logarithmic transform to perform convolution operations on the data to enhance the edges.

[0098] Furthermore, in step S2, principal component analysis is used for dimensionality reduction, and the dimensionality reduction formula is as follows:

[0099]

[0100] Where E is the PCA transformation matrix, It is the mean spectrum; the PCA model parameters 𝐸 and mean spectrum 𝜇 are obtained by jointly fitting the preprocessed hyperspectral data of the target and background samples in the training set;

[0101] The dimensionality-reduced data is arranged according to spatial dimensions to obtain K-channel principal component feature maps, and pseudo-color images are formed by grouping every three channels. When K is not an integer multiple of 3, the last group is supplemented to three channels by copying or padding.

[0102] Furthermore, in step S3, the fusion employs Alpha mixing, with the following formula:

[0103]

[0104] Where α=0.9, and the transparency factor α∈[0.85,0.95], we obtain 𝑀 true-to-life pseudo-color images. .

[0105] Further, in step S4, the generator of the improved StyleGAN2-ADA model includes a mapping network and a synthesis network. The conditional control parameters and random noise are input into the mapping network to generate a conditional style vector. The synthesis network includes a self-attention module for capturing the global correlation between the target and the background.

[0106] Furthermore, in step S4, the loss functions used in adversarial training include non-saturated adversarial loss, path length regularization loss, and RGB channel consistency loss, wherein the RGB channel consistency loss is constructed based on the Pearson correlation coefficient between each channel of the generated image and the real image.

[0107] Further, in step S5, the three channels of each group forming the pseudo-color image are restored to their corresponding partial K-channel data through linear mapping, and the complete K-channel synthetic principal component feature map is obtained after splicing all groups of data; the formula for the inverse PCA transform is:

[0108]

[0109] in, It is the recovered original spectrum. Synthesis is the K-channel principal component features of each pixel, μ is the mean spectrum during PCA training, and 𝐸 is the transformation matrix of PCA, which transforms K-dimensional data into N-dimensional data.

[0110] Furthermore, step S6 specifically includes:

[0111] S61: The synthesized hyperspectral data cube is processed by the spatial attention module to generate an attention weight map highlighting the target region, and the weight map is multiplied with the original data to obtain the enhanced data;

[0112] S62: Input the enhanced data into the three-dimensional convolutional neural network detection model to extract spatial-spectral joint features and predict the target bounding box and category;

[0113] S63: Perform non-maximum suppression on all predicted target bounding boxes, filter out overlapping boxes, and output the final target detection result.

[0114] Example 2:

[0115] A method for sparse small target data enhancement and background fusion in hyperspectral images is proposed. Through five core steps—data preprocessing adaptation, model architecture modification, introduction of conditional generation mechanisms, spectral domain ADA-enhanced training, and seamless target fusion—it achieves "spectral fidelity, spatial consistency, and target controllability" in small-sample scenarios. The technical process is as follows: Figure 1 As shown, the implementation steps are as follows:

[0116] Step 1: Hyperspectral data preprocessing:

[0117] The hyperspectral data containing the target and the background are used as input. Laplacian transform, logarithmic transform and image scaling are performed to enhance the edge details of the target and compress the dynamic range of the data, resulting in "preprocessed hyperspectral segmentation data (hyperspectral data of the target and background)".

[0118] 1.1 Logarithmic Transformation: A logarithmic transformation is performed on the normalized hyperspectral data, as shown in Formula (1.1), to enhance spectral details in low reflectivity regions:

[0119] ;

[0120] in, =1e-8, For the normalized data, This is the transformed data. The operation is performed band-by-band on all pixels, preserving spectral information while enhancing the visibility of the target area.

[0121] 1.2 Laplace Transform: For the logarithmically transformed data, edge enhancement is performed using a 3×3 Laplace operator (kernel) (1.2) to highlight the boundary difference between the target and the background. The Laplace kernel is:

[0122]

[0123] ;

[0124] in, For pixel coordinates, This is the value after the Laplace transform.

[0125] 1.3 Image Scaling (Uniform Size): The spatial dimensions of the original hyperspectral data are scaled to fit the subsequent network input. Bilinear interpolation is used to scale the 2D image for each band individually: the scaling factor between the original size and the target size (256×256) is calculated; for each target pixel location, the corresponding pixel value in the original image is calculated through interpolation to ensure that the spatial structure is not distorted after scaling. Output: Preprocessed hyperspectral segmentation data of 256×256×N, while retaining the segmentation labels for both the target and background.

[0126] Step 2: PCA dimensionality reduction compresses N-band data into K channels:

[0127] 2.1 PCA Dimensionality Reduction: To ensure sufficient preservation of high-dimensional spectral information, this embodiment employs an advanced adaptive dimensionality reduction method. Principal Component Analysis (PCA) or other adaptive dimensionality reduction methods (such as autoencoders, t-SNE, etc.) are used to reduce the dimensionality of the hyperspectral data of the target and background, mapping the spectral vector of each pixel to a low-dimensional representation. PCA combined with band selection avoids information loss that may occur in traditional methods. First, PCA dimensionality reduction is performed on the hyperspectral data of the target and background, mapping the spectral vector of each pixel to a low-dimensional representation. Mapping to a low-dimensional representation ,in ;

[0128] ;

[0129] Where E is the PCA transformation matrix, It is the mean spectrum. The PCA model parameter E mean spectrum. It is obtained by jointly fitting preprocessed hyperspectral data of target and background samples in the training set.

[0130] 2.2 Constructing a pseudo-color image: The dimensionality-reduced hyperspectral data 𝑦∈𝑅 256×256×𝐾 Grouped into groups of 3 principal component channels each, with the number of groups being [number missing]. When K is not a multiple of 3, the missing channels in the last group are padded by copying the previous channel or using zero-padding to ensure that all pseudo-color images have 3 channels as input. Based on the segmentation labels, the target region and mask are extracted, and each group of targets is mapped to a target pseudo-color image. A schematic diagram of pseudo-color RGB segmentation is shown below. Figure 2 As shown, the horizontal axis represents the number of spectral channels (1-N), and the vertical axis represents the wavelength (nm). Different color blocks are used to mark the channel ranges corresponding to the R / G / B channels (e.g., R=150-N, G=80-149, B=1-79), visually demonstrating the segmentation logic.

[0131] Step 3: Target-Background Blending (blending group by group in the pseudo-color domain to obtain M true-blended pseudo-color images)

[0132] 3.1 Randomly select the target placement position on the background pseudo-color RGB image (avoiding image edges to ensure target integrity), with the position coordinates (x0, y0) following a uniform distribution;

[0133] The target pseudo-color RGB image is scaled to match the spatial dimensions of the background objects. This adapts to the needs of generating sparse, small targets. Transparency blending: alpha blending is used, with the formula:

[0134]

[0135] Where α = 0.9 (target transparency, ensuring that target features are not obscured by the background), transparency factor α ∈ [0.85, 0.95], resulting in M ​​true-to-false-color fused images. .

[0136] 3.2 Edge Blending: An alpha blending method is used to blend the target image and the background image to ensure a smooth transition between the target and the background and avoid hard cutting. Transparency value. The setting is crucial; in this invention, the transparency of the target is set to... =0.9, the purpose of which is to ensure that the target features are clearly visible in the background and not excessively obscured by the background. The specific fusion formula is as follows:

[0137] ;

[0138] in, For the final merged image, and These are the pixel values ​​of the target and background images at their respective locations, and α=0.9 is the transparency coefficient of the target image, ensuring that the target image is displayed clearly and naturally in the background.

[0139] Step 4: StyleGAN2-ADA Generates Pseudo-Color Groups: Input the pseudo-color group image and "Generate Image Labels, Conditions" (target category, quantity, location, and other control parameters). Send the input data to the StyleGAN2-ADA generator. The network structure diagram of StyleGAN2-ADA is shown below. Figure 4 As shown, a fused image containing sparse small targets is generated by combining labels and conditions; the discriminator simultaneously distinguishes between real and fake data in the "real overlay data" and the "generated fused image", driving the generator to optimize the synthesis quality. The output is the generated fused image (pseudo-color group data containing sparse small targets).

[0140] 4.1 Generator Process:

[0141] The generator consists of a mapping network and a synthesis network, with the attention module in the synthesis network being crucial. Labels (target / background categories) and random noise are input into the mapping network (3 fully connected layers, FC), generating a 512-dimensional latent vector.

[0142] The synthetic network (including an attention module) starts at a 4×4 resolution and progressively enlarges the image through upsampling (4×4→8×8→…→256×256). After each upsampling round, convolutional layers extract features, while a self-attention module is inserted to capture the global correlation between the target and the background. Finally, a "toRGB layer" converts the high-dimensional features into a 256×256×3 pseudo-color image. This ultimately generates M sets of pseudo-color fused images.

[0143] 4.2 Discriminator Process:

[0144] The discriminator is responsible for distinguishing between "realistic fused images" and "generated fused images." The preprocessed target, background overlay, and labels are input into the discriminator and converted into feature maps using the "fromRGB" module. The "Down" module progressively downsamples the images (256×256 → ...), while skip connections preserve multi-scale features. After "Mbstdnormalization" (multi-scale batch normalization), a fully connected layer (FC) outputs the true / false probabilities (0 = generated image, 1 = real image).

[0145] 4.3 Combat Training:

[0146] The generator and discriminator are optimized adversarially using a loss function: Generator loss: causes the discriminator to misclassify generated images as "real"; Discriminator loss: correctly distinguishes between real and generated images. Training terminates when the validation set FID (FréchetInceptionDistance) < 30, ensuring that the distribution of generated images is consistent with that of real images.

[0147] 4.4 Experimental Setup:

[0148] Hardware environment: Processor: Intel Xeon CPU 2.4GHz; GPU: NVIDIA Tesla A100, 40GB VRAM; Memory: 128GB RAM; Storage: 2TB SSD; Operating system: Ubuntu 20.04.

[0149] The superimposed hybrid pseudo-color RGB images were divided into a training set (90%) and a validation set (10%), and converted to TFRecords format (compatible with TensorFlow's StyleGAN2-ADA). The batch size was set to 8 (for a single card with 12GB of VRAM). The loss function used was non-saturating adversarial loss + path length regularization loss (λ_pl=0.01), and a new "RGB channel consistency loss" was added (to ensure that the channel correlation of the generated images matches the hyperspectral characteristics).

[0150] ;

[0151] in, To generate the RGB channels of the image, For the RGB channels of a true pseudo-color RGB image, Pearson correlation coefficient; Optimizer: Adam (β1=0.0, β2=0.99, ε=1e-8), generator learning rate 0.001, discriminator learning rate 0.002; ADA enhancement strategy: retain only geometric transformations (flip, rotate, crop), disable color transformations (to avoid destroying the spectral correspondence of pseudo-color), and adaptively adjust the enhancement probability (0-0.8); Training termination condition: validation set FID<30, training iterations 200,000 steps. After training convergence, input a random noise vector z (512-dimensional) into StyleGAN2-ADA to generate multiple sets of pseudo-color RGB images with different principal component channel groups: Output: Each set generates 10,000 pseudo-color RGB images containing sparse small targets, covering different target numbers, positions, sizes, and angles.

[0152] Step 5: Generate pseudo-color features of the K-channel principal components from multiple sets of data, and then use inverse PCA to restore them to the N-band.

[0153] 5.1 Recovery of pseudo-color group to K-channel principal component feature map:

[0154] In step S4, M sets of pseudo-color images generated using StyleGAN2-ADA are used, with 10,000 images generated in each set. Each image is 256×256×3 pixels in size and includes the synthesized target and background. Three channels are extracted from each pseudo-color image and mapped to the original K-channel principal component feature map. When K is not a multiple of 3, the missing channels in the last set are padded by copying the previous channel or using zero-padding to ensure that all pseudo-color images have 3-channel input. ;

[0155] Three RGB channels are extracted from each pseudocolor image. The values ​​of these three channels are then mapped to the original K-channel principal component feature map. A linear transformation is typically used when mapping the RGB channels of a pseudocolor image to the K-channel principal component feature map. The core idea of ​​this linear mapping is to map the spectral information of the RGB channels to a K-dimensional space through a weighted summation, making the mapped data as close as possible to the dimensionality-reduced principal component feature map. Figure 1 The linear mapping is performed based on the RGB pixel values ​​and the principal component transformation matrix obtained during PCA dimensionality reduction. Since PCA reduces dimensionality by preserving the main features of the data, this linear transformation can maintain the continuity and consistency of the spectrum to the greatest extent. When K is not an integer multiple of 3, the previous channel is copied or zero-padding is performed to ensure that each pseudo-color image has K channels, so as to align with the principal component feature map of the hyperspectral data.

[0156] 5.2 PCA inverse transform to recover N-band hyperspectral images:

[0157] K-channel principal component feature map To restore the original N-band hyperspectral data, perform an inverse PCA transform:

[0158] ;

[0159] In the formula, It is the recovered original spectrum. Synthesis is the K-channel principal component features of each pixel, μ is the mean spectrum during PCA training, and E is the transformation matrix of PCA, which transforms K-dimensional data into N-dimensional data.

[0160] By performing inverse PCA on the K-channel features of each pixel, the spectral features of the corresponding N bands are recovered pixel by pixel. The final result is a recovered hyperspectral data cube.

[0161] ;

[0162] In the formula, n∈{1,2,…,N} is the band index. These are elements in the PCA transform matrix, representing the relationship between the nth band and the kth principal component. It is the value of the Kth principal component.

[0163] Comparison of spectral reconstruction effects is shown in the figure below. Figure 3 As shown, the line graph compares the "real hyperspectral spectrum curve", "single set of pseudo-color RGB reconstruction curve" and "multiple sets of pseudo-color RGB reconstruction curve", and marks the SAM value (e.g., SAM=0.06 for multiple reconstructions), verifying the reconstruction effect.

[0164] Step 6: Detection and verification of the synthesized data;

[0165] The input is the synthesized hyperspectral data cube recovered through step 5. First, attention enhancement is performed, then the input is fed into the detection network for target recognition; finally, duplicate detection boxes are filtered out through NMS post-processing (non-maximum suppression) to obtain the final detection result.

[0166] 6.1 Attention mechanisms enhance target features:

[0167] The generated image is input into the spatial attention module, and global average pooling is performed on the image to obtain channel attention weights. A 256×256×1 attention map is generated through two fully connected layers (target region weight ≥ 0.8, background region weight ≤ 0.5). The attention map is multiplied pixel by pixel with the generated image to highlight the target region in the generated image and suppress background noise.

[0168] 6.2 Detection network identifies targets:

[0169] The "target-enhanced image" is input into a 3D-CNN detection network (with a 3D structure adapted to hyperspectral data). The network extracts spatial spectral features through "convolution + pooling" to predict the target's location (boundingbox) and category.

[0170] 6.3 NMS Post-processing (Non-maximum suppression):

[0171] For all detection boxes output by the detection network, sort them by confidence level; select the box with the highest confidence level in turn, and delete other boxes with an intersection-over-union (IoU) ratio (IoU) greater than 0.5; finally output the "detection result" (including the location, category, and confidence level of the target).

[0172] Experimental results:

[0173] In the experiments, standard hyperspectral datasets were used for comparison, including the Pavia University dataset and the AVIRIS dataset. These datasets contain hyperspectral images in multiple bands and are widely used in hyperspectral image analysis and object detection tasks. The method of this invention was compared with the existing CascadeGAN algorithm by evaluating the quality of the generated images and the downstream object detection performance on these datasets. By evaluating metrics such as FID (Fréchet Inception Distance) and SSIM (Structural Similarity Index) of the generated images, the method of this invention showed superior image quality, with an FID score of 29.4, significantly lower than CascadeGAN's 38.7, and an SSIM value of 0.92, higher than CascadeGAN's 0.85 and 0.88. Furthermore, the accuracy of this invention in the downstream object detection task reached 85%, compared to CascadeGAN's 70%. These results demonstrate that this invention not only outperforms existing technologies in image generation quality but also significantly improves the performance of downstream tasks, especially in the field of sparse small object detection, reflecting the effectiveness of the generated data in practical applications.

[0174] Core strengths and key innovations:

[0175] 1. Complete preservation of spectral information: By reconstructing pseudo-color RGB images segmented from multiple sets of different spectral bands, the loss of spectral information caused by single RGB segmentation is avoided, ensuring the spectral continuity of the generated data;

[0176] 2. Naturalness of target fusion: The target and background are superimposed first and then input into the model for training, resulting in a fusion effect that is closer to the real hyperspectral scene.

[0177] The core logic of this solution can be summarized in three key steps:

[0178] 1. The hyperspectral data is dimensionality reduced to obtain principal component channels. The principal component feature channels are grouped into groups of three to obtain pseudo-color RGB images, which are adapted to the 3-channel input of StyleGAN2-ADA.

[0179] 2. Based on the superimposed target-background pseudo-color images, a training model is developed to generate multiple sets of high-quality pseudo-color RGB samples;

[0180] 3. By restoring multiple sets of RGB samples to principal component channels and then performing inverse PCA transformation to complete the hyperspectral data cube, both generation efficiency and spectral fidelity are balanced.

[0181] The entire process requires no modification to the StyleGAN2-ADA core code; it adapts to hyperspectral scenarios only through data preprocessing and post-reconstruction, making it easy to implement and deploy.

[0182] Analysis of the principle behind the technical problem solved by this invention (combining the solution and the inventive point):

[0183] (1) The compatibility problem between high-dimensional spectral data and the 2DGAN framework is solved by using the strategy of "band dimensionality reduction + channelization".

[0184] This invention employs Principal Component Analysis (PCA) or Minimum Noise Separation (MNF) for band dimensionality reduction during the data preprocessing stage, retaining core bands with a cumulative contribution rate ≥95% (e.g., 64 bands). This removes redundant information, reduces computational burden, and maintains high inter-spectral correlation. Simultaneously, the N dimensionality-reduced bands are converted into an N-channel tensor format, adapting the hyperspectral data cube (H×W×λ) to the input requirements of 2D convolutional networks. This design directly addresses the model training collapse problem caused by the curse of dimensionality and, through information preservation strategies during dimensionality reduction, ensures that the spectral curves of the generated samples are smooth and continuous, conforming to physical laws.

[0185] (2) By using "dual condition injection + feature balancing design", the problem of imbalance between fine control and feature learning of sparse small targets is solved.

[0186] This invention transforms category labels into 512-dimensional features through embedding vectors. These features are then concatenated with random noise vectors and injected into a mapping network. Furthermore, affine parameters generated by a small MLP are injected into each AdaIN layer of the synthesis network, achieving basic control over the presence or absence of targets. Advanced schemes can also incorporate conditions such as Gaussian heatmaps and pose angles to achieve fine-grained control over the number, location, and pose of targets. Simultaneously, by constructing a target sample library and a background sample library (with a ratio of 1:3-5), and strengthening the modulation weight of target features during conditional generation, the generator avoids ignoring the weak features of sparse small targets due to an excessively high proportion of background pixels. This balances the learning priorities of background and target, solving the problems of low accuracy and poor recognition in the generation of sparse small targets.

[0187] (3) By using the “generation-extraction-learning fusion” process, the problem of seamless fusion of target and background and spatial-spectral consistency is solved.

[0188] This invention first uses a segmentation network to accurately extract the spectral and shape information of the generated target. Then, it utilizes a small U-Net fusion network, taking "pure background + target spectral patch + location mask" as input, and trains it using L1 loss and spectral angle matching (SAM) loss to achieve a natural fusion of the target and background, generating realistic shadows and edge transitions. This design avoids the abrupt edge problems caused by simple pasting. At the same time, the training process of the fusion network forcibly constrains the synergy between spatial texture and spectral features, ensuring the spatial continuity of the spectral curves of objects in the same area, maintaining the spatial-spectral consistency of the generated scene, and meeting the requirements of downstream detection tasks for sample authenticity.

[0189] Additional technical effects achieved by this invention:

[0190] (1) The diversity and controllability of generated samples are significantly improved.

[0191] By dynamically adjusting the spectral enhancement intensity, randomly selecting the target location and attitude, and controlling the target embedding density (embedding 5-20 targets per 100 background samples), the generated samples can cover scenes with different land cover types, lighting conditions, and target states, with a diversity far exceeding that of existing fixed generation schemes. Furthermore, through the conditional injection mechanism, samples with "pure background," "with targets," or specific parameters (quantity, location, attitude) can be generated on demand, adapting to the diverse training needs of downstream detection models.

[0192] (2) Optimization of training efficiency and resource adaptability

[0193] The design adopts "transfer learning + adjustable network capacity". Fine-tuning of large-scale hyperspectral pre-trained models can shorten the convergence time by 30%-50%. The number of channels and convolutional blocks of the generator / discriminator can be flexibly adjusted according to the GPU memory (e.g., the basic number of channels is maintained when the single card has 8GB, and doubled when it has 24GB), which reduces the threshold of computing resources and adapts to the hardware conditions of different research institutions.

[0194] (3) The performance improvement effect of downstream tasks can be quantified.

[0195] The generated samples undergo a three-pronged evaluation based on "general generation quality (FID / SSIM), spectral fidelity (SAM / RMSE / ERGAS), and downstream task performance" to ensure the validity of the synthetic data. In practical applications, training the target detection model by mixing synthetic samples with real samples can improve the detection rate by ≥15% and reduce the false alarm rate by ≥10%, significantly overcoming the performance bottleneck caused by the scarcity of real labeled samples.

[0196] (4) Interpretability of spectral features and potential for expanded applications

[0197] StyleGAN2-ADA's W latent space can decouple the physical properties of hyperspectral data (such as vegetation health, soil moisture, and light intensity). Through linear interpolation and principal component analysis, the corresponding control vectors can be identified, making it possible to generate "digital twin" hyperspectral data (such as vegetation samples with different water contents), thus expanding the application scenarios of the technology in precision agriculture, environmental monitoring, and other fields.

[0198] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for enhancing sparse small target data and fusing background in hyperspectral images, characterized in that, Includes the following steps: Step S1: Hyperspectral data preprocessing. Logarithmic transformation and Laplacian transformation are performed on the input hyperspectral data containing the target and background to enhance details, and image scaling is performed to unify spatial size, resulting in preprocessed hyperspectral segmentation data. Step S2: Dimensionality reduction is performed on the preprocessed hyperspectral segmentation data, the high-dimensional band data is compressed into a K-channel principal component feature map, and the K-channel principal component feature map is divided into M groups of three-channel pseudo-color images; Step S3: Fuse the pseudo-color image representing the target with the pseudo-color image representing the background in the pseudo-color domain to generate multiple realistic fused pseudo-color training images; Step S4: Input the real fused pseudo-color training image and its corresponding conditional control parameters into the improved StyleGAN2-ADA model for adversarial training. The conditional control parameters include at least target category, quantity, and location information. After training, the improved StyleGAN2-ADA model generates a set of synthetic pseudo-color images containing sparse small targets. Step S5: Map the synthesized pseudo-color image group back to a K-channel synthesized principal component feature map, and perform inverse principal component analysis on the K-channel synthesized principal component feature map to regenerate an N-band synthesized hyperspectral data cube. Step S6: Perform target detection verification on the synthesized hyperspectral data cube and output the detection results.

2. The method according to claim 1, characterized in that, In step S1, the formula for the logarithmic transformation is: in, =1e-8, For the normalized data, The transformed data; The Laplace transform uses a 3×3 Laplace kernel logarithmic transform to perform convolution operations on the data to enhance the edges.

3. The method according to claim 1, characterized in that, In step S2, principal component analysis is used for dimensionality reduction. The dimensionality reduction formula is as follows: Where E is the PCA transformation matrix, It is the mean spectrum; the PCA model parameters 𝐸 and mean spectrum 𝜇 are obtained by jointly fitting the preprocessed hyperspectral data of the target and background samples in the training set; The dimensionality-reduced data is arranged according to spatial dimensions to obtain K-channel principal component feature maps, and pseudo-color images are formed by grouping every three channels. When K is not an integer multiple of 3, the last group is supplemented to three channels by copying or padding.

4. The method according to claim 1, characterized in that, In step S3, the fusion adopts Alpha mixing, and the formula is: Where α=0.9, and the transparency factor α∈[0.85,0.95], we obtain 𝑀 true-to-life pseudo-color images. .

5. The method according to claim 1, characterized in that, In step S4, the generator of the improved StyleGAN2-ADA model includes a mapping network and a synthesis network. The conditional control parameters and random noise are input into the mapping network to generate a conditional style vector. The synthesis network includes a self-attention module for capturing the global correlation between the target and the background.

6. The method according to claim 5, characterized in that, In step S4, the loss functions used in adversarial training include non-saturated adversarial loss, path length regularization loss, and RGB channel consistency loss. The RGB channel consistency loss is constructed based on the Pearson correlation coefficient between each channel of the generated image and the real image.

7. The method according to claim 1, characterized in that, In step S5, the three channels of each group forming the pseudo-color image are restored to their corresponding partial K-channel data through linear mapping. After stitching together all groups of data, a complete K-channel synthetic principal component feature map is obtained. The formula for the inverse PCA transform is: in, It is the recovered original spectrum. Synthesis is the K-channel principal component features of each pixel, μ is the mean spectrum during PCA training, and 𝐸 is the transformation matrix of PCA, which transforms K-dimensional data into N-dimensional data.

8. The method according to claim 1, characterized in that, Step S6 specifically includes: S61: The synthesized hyperspectral data cube is processed by the spatial attention module to generate an attention weight map highlighting the target region, and the weight map is multiplied with the original data to obtain the enhanced data; S62: Input the enhanced data into the three-dimensional convolutional neural network detection model to extract spatial-spectral joint features and predict the target bounding box and category; S63: Perform non-maximum suppression on all predicted target bounding boxes, filter out overlapping boxes, and output the final target detection result.

Images