Hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perception gradient loss

Abstract

This invention discloses a hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss, comprising the following steps: collecting RGB and hyperspectral images; transmitting the collected images to a computer for preprocessing; summarizing the preprocessed data to generate an image fusion dataset; inputting the dataset into a model; generating pseudo-images through convolution, encoding, and decoding via a primary processing module and a generator in the model; feeding the pseudo-images into a discriminator in the model; analyzing the differences between the pseudo-images and real images; calculating the total perceptual gradient loss between the pseudo-images and real images using the total perceptual gradient loss function; evaluating the similarity between the two in the feature space; and feeding the obtained loss back to the generator to optimize the generation process; if the discriminator determines the image is real, it outputs the result; if it determines it is fake, it feeds back to the generator for continued training; using this invention, RGB and hyperspectral images can be fused completely, accurately, and quickly.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint sensing gradient loss. Background Technology

[0002] Crop diseases and pests, characterized by their diverse occurrences, wide impact, and the potential for localized outbreaks, have long been a significant factor restricting agricultural production, substantially affecting crop yield and quality. In recent years, the occurrence and prevalence of diseases and pests have intensified, making their control even more challenging. According to estimates by the Food and Agriculture Organization of the United Nations (FAO), diseases and pests cause approximately one-quarter of the world's annual crop yield loss, with diseases accounting for 14% and insects for 10%. Therefore, accurately and timely acquiring information on the spatial distribution and development trends of diseases and pests in farmland is crucial for disease control. Current advancements in image processing technology and the widespread availability of RGB cameras have made it easy to acquire RGB images, and the effective information captured by RGB images is playing a vital role in crop disease and pest detection. However, RGB images primarily contain information from the red, green, and blue channels. While they can replace manual inspection and accurately provide information on disease and pest infestation, they are incapable of capturing the more refined spectral characteristics of diseases and pests. Therefore, hyperspectral imaging has emerged as a solution. Hyperspectral images express specific scene information through hundreds or thousands of consecutive narrow bands. This information not only contains rich spatial information but also spectral information absent in RGB images, enabling more precise monitoring of crop pests and diseases. However, hyperspectral images have limited spectral information representation, sometimes failing to fully express all features. Furthermore, capturing three-dimensional data cubes using spectral or spatial scanning methods often requires significant exposure time. Sacrificing temporal resolution for high-quality spectral resolution hinders the application of hyperspectral images in crop pest and disease monitoring. Therefore, there is an urgent need for a method that can completely, accurately, and rapidly fuse RGB and hyperspectral images. Summary of the Invention

[0003] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0004] In view of the problems existing in the above and / or existing image processing techniques, the present invention is proposed.

[0005] Therefore, the problem that this invention aims to solve is that the existing technology for fusing RGB and hyperspectral images is slow and cannot fully express all features. Using this invention, RGB and hyperspectral images can be fused completely, accurately, and quickly.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint sensing gradient loss, comprising the following steps:

[0007] Collect RGB and hyperspectral images, transfer the collected images to a computer, preprocess the images, summarize the preprocessed data, and generate an image fusion dataset.

[0008] The image fusion dataset is input into the SA-DAEDCGAN model. Through the primary processing module and the generator in the SA-DAEDCGAN model, pseudo images are generated through convolution, encoding, and decoding. The pseudo images are then fed into the discriminator in the SA-DAEDCGAN model. The discriminator analyzes the differences between the pseudo images and the real images. The pseudo images and the real images are compared using the total perceptual gradient loss function L(G,D) to calculate the total perceptual gradient loss, which evaluates the similarity between the two in the feature space. The obtained loss is fed back to the generator to optimize the generation process. If the discriminator determines that the image is real, it outputs the result; if it determines that it is fake, it feeds back to the generator to continue training.

[0009] As a preferred embodiment of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in this invention, the generator introduces a denoising autoencoder network architecture in the input layer to encode and denoise the input hyperspectral and RGB images, extract and integrate features, and inject them into the new image to achieve fusion.

[0010] As a preferred embodiment of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in this invention, wherein: the discriminator incorporates a self-attention mechanism in its structure.

[0011] As a preferred embodiment of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint sensing gradient loss in this invention, the method involves the following steps during preprocessing: for hyperspectral images, the one-dimensional average reflectance curve of the region of interest is first extracted using ENVI, and then the one-dimensional feature curve data is converted into two-dimensional feature image data using the Auto-CWT method, ultimately forming an irrelevant feature dataset.

[0012] As a preferred embodiment of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint sensing gradient loss in this invention, the Auto-CWT method incorporates an autoencoder into the continuous wavelet transform method, as shown in the following formula.

[0013] z = f encoder (CWT(x(t)))

[0014] W x (a,b)=f decoder (z)

[0015]

[0016] Where z represents the low-dimensional space, f encoder The mapping function f of the encoder. decoder The decoder's mapping function is represented by CWT(x(t)), which represents the original CWT feature matrix, and W... x (a,b) represents the reconstruction matrix, and L represents the loss function.

[0017] As a preferred embodiment of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in this invention, the primary processing module first upsamples the RGB image to obtain image information with the same spatial size as the hyperspectral image, and then extracts shallow information from the RGB and hyperspectral images through two convolutional layers and maps them to the same spectral dimension.

[0018] As a preferred embodiment of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint sensing gradient loss in this invention, wherein: the formula for the total sensing gradient loss function is L(G,D)=αL GAN +βL Con +λL PG .

[0019] Among them, L GAN L represents the adversarial loss function; Con L represents the image content loss function; PG Let represent the subsensory gradient loss function; α, β, and λ represent learnable adaptive weight coefficients.

[0020] As a preferred embodiment of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in this invention, wherein: the formula for the adversarial loss function is,

[0021]

[0022] Where x represents the real image x, G(z) represents the image generated by the generator, and Ex～pdata(x) For the true image distribution P data The expected value of P on (x), data (x) represents the probability distribution of the real image. To generate an image in P data The expected value on (z), P data (z) represents the probability distribution of the image generated by the generator, D(x) represents the probability output of the discriminator that the input image is a real image, and D(G(z)) represents the probability output of the discriminator that the image generated by the generator is a real image.

[0023] As a preferred embodiment of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in this invention, the image content loss function measures the similarity between the generated image and the target image by calculating the difference in high-level semantic features, and the formula is:

[0024]

[0025] Where H and W represent the height and width of the image, i is the row index of a pixel in the image, j is the column index of a pixel in the image, and x... ij Let G(z) be the pixel value of the real image at coordinates (i, j). ij To generate the pixel value of image G(z) at coordinates (i, j).

[0026] As a preferred embodiment of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in this invention, the formula for the perceptual gradient loss function is:

[0027] L PG =λ1L P +λ2L G

[0028]

[0029] Among them, L P L represents perceived loss. G Represents gradient loss, These are represented as horizontal and vertical gradient operators, respectively.

[0030] The beneficial effects of this invention are as follows: This invention integrates the advantages of generative adversarial networks, self-attention mechanisms, and denoising autoencoders, proposing an SA-DAEDCGAN model that fuses hyperspectral and RGB images. A primary processing module and a denoising autoencoder architecture are introduced into the generator, enabling the network to simultaneously perceive local and global features. A discriminator with a self-attention mechanism is constructed to ensure the quality of the reconstructed image. A novel perceptual gradient loss function is proposed, improving the model's training stability and generating higher-quality, more realistic images. Furthermore, a novel Auto-CWT image conversion method is proposed to address the problem of weakened regions of interest in fused images in some monitoring fields, improving recognition accuracy. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0032] Figure 1 This is a schematic diagram of the process of the present invention.

[0033] Figure 2 This refers to the Auto-CWT preprocessing procedure in this invention.

[0034] Figure 3 This is a schematic diagram of the generator in the model.

[0035] Figure 4 This is a schematic diagram of the discriminator in the model.

[0036] Figure 5 The comparison charts are of the evaluation results of the performance comparison experiment in Example 2 ((a) is a stacked chart of the evaluation results, and (b) is a bar chart of the evaluation results).

[0037] Figure 6 This is a heatmap comparing the evaluation results of the performance comparison experiment in Example 2.

[0038] Figure 7 The following figures show the results of the accuracy experiment in Example 2: (a) Model training accuracy curve, (b) Model training loss curve, (c) Confusion matrix heatmap of the validation set results of the fused dataset, and (d) Confusion matrix heatmap of the validation set results of the original dataset.

[0039] Figure 8 The following are comparison charts of the evaluation results of the performance comparison experiment in Example 3 ((a) is a stacked chart of the evaluation results, and (b) is a bar chart of the evaluation results).

[0040] Figure 9 This is a heatmap comparing the evaluation results of the performance comparison experiment in Example 3. Detailed Implementation

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0042] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0043] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0044] Example 1

[0045] Reference Figures 1-4 This is the first embodiment of the present invention, which provides a hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint sensing gradient loss, which can completely, accurately and quickly fuse RGB and hyperspectral images.

[0046] A hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint sensing gradient loss includes the following steps:

[0047] Step 1, Data Acquisition and Preprocessing: Collect RGB images and hyperspectral images, transfer the collected images to the computer, and preprocess the images. During preprocessing, for hyperspectral images, firstly, extract the one-dimensional average reflectance curve of the region of interest using ENVI, and then use the Auto-CWT method to convert the one-dimensional feature curve data into two-dimensional feature image data, finally forming an irrelevant feature dataset. Summarize the preprocessed data to generate an image fusion dataset.

[0048] The Auto-CWT method incorporates an autoencoder into the continuous wavelet transform method, as shown in the following formula.

[0049] z = f encoder (CWT(x(t)))

[0050] W x (a,b)=f decoder (z)

[0051]

[0052] Where z represents the low-dimensional space, f encoder The mapping function f of the encoder. decoder The decoder's mapping function is represented by CWT(x(t)), which represents the original CWT feature matrix, and W... x (a,b) represents the reconstruction matrix, and L represents the loss function;

[0053] Step 2, Model Building and Optimization: The image fusion dataset is input into the SA-DAEDCGAN model. Through the primary processing module and the generator in the SA-DAEDCGAN model, pseudo images are generated through convolution, encoding, and decoding. The pseudo images are then fed into the discriminator in the SA-DAEDCGAN model. The discriminator analyzes the differences between the pseudo images and the real images. The pseudo images and the real images are compared using the total perceptual gradient loss function L(G,D) to calculate the total perceptual gradient loss, assess the similarity between the two in the feature space, and feed the obtained loss back to the generator to optimize the generation process. If the discriminator determines that the image is real, it outputs the result; if it determines that it is fake, it feeds back to the generator to continue training.

[0054] The primary processing module first upsamples the RGB image to obtain image information with the same spatial size as the hyperspectral image. Then, it extracts shallow information from the RGB and hyperspectral images through two convolutional layers and maps them to the same spectral dimension.

[0055] The generator introduces a denoising autoencoder network architecture into the input layer to encode and denoise the input hyperspectral and RGB images, extract and integrate features, and inject them into the new image to achieve fusion, such as... Figure 3 As shown, Conv1 to Conv4 are all convolutional layers; the network between Conv3 and Conv4 is the introduced denoising autoencoder architecture; the main feature processing involves extracting shallow information from RGB and hyperspectral data through two convolutional layers and mapping it to the same spectral dimension; the discriminator incorporates a self-attention mechanism into the structure, combining the self-attention mechanism with the convolutional layers to enhance the ability to distinguish between genuine and fake images, such as... Figure 4As shown, the image passes through three convolutional layers (Conv1, Conv2, Conv3) sequentially. These convolutional layers are used for feature extraction, progressively capturing complex spatial features in the image. After the convolutional layers, the architecture includes a self-attention mechanism, which works in conjunction with an additional convolutional layer (Conv4). The self-attention mechanism helps the network focus on important parts of the image, capturing long-distance dependencies between image pixels, and thus selecting more relevant features to make judgments based on the weights of different regions of the image. After the self-attention layer, the output passes through a softmax layer (normalized exponential function layer) to generate a probability distribution to determine whether the input image is more likely to be real or generated. Finally, the discriminator's output is a binary classification result: the input image is classified as fake or real.

[0056] The formula for the total perceived gradient loss function is: L(G,D)=αL GAN +βL Con +λL PG .

[0057] Among them, L GAN L represents the adversarial loss function; Con L represents the image content loss function; PG Let represent the subsensory gradient loss function; α, β, and λ represent learnable adaptive weight coefficients.

[0058] Adversarial loss function L GAN By minimizing the model's loss on adversarial examples, the robustness of the model to input perturbations is improved. The formula is as follows:

[0059]

[0060] Where x represents the real image x, and G(z) represents the image generated by the generator. For the true image distribution P data The expected value of P on (x), data (x) represents the probability distribution of the real image. To generate an image in P data The expected value on (z), P data (z) represents the probability distribution of the image generated by the generator, D(x) represents the probability output of the discriminator that the input image is a real image, and D(G(z)) represents the probability output of the discriminator that the image generated by the generator is a real image.

[0061] Image content loss function L Con The similarity between the generated image and the target image is measured by calculating the difference in high-level semantic features. The formula is as follows:

[0062]

[0063] Where H and W represent the height and width of the image, i is the row index of a pixel in the image, j is the column index of a pixel in the image, and x... ij Let G(z) be the pixel value of the real image at coordinates (i, j). ij To generate the pixel value of image G(z) at coordinates (i, j).

[0064] The sensing gradient loss function L PG By weighted summing of the perceptual loss and gradient loss, we can simultaneously consider the performance of the generated image in both the feature space and gradient edge information, thereby improving the visual quality and detail preservation of the image. The formula is as follows:

[0065] L PG =λ1L P +λ2L G

[0066]

[0067] Among them, L P L represents perceived loss. G Represents gradient loss, These are represented as horizontal and vertical gradient operators, respectively.

[0068] This application significantly improves the fusion effect of RGB and hyperspectral images by introducing the SA-DAEDCGAN model. This model innovatively combines generative adversarial networks, self-attention mechanisms, and denoising autoencoders, solving the technical bottlenecks of limited spectral information in single RGB images and low resolution in hyperspectral images. It overcomes the problems of existing fusion algorithms losing important features and generating low-quality images. Specifically, the primary processing module and denoising autoencoder architecture added to the generator enable the network to simultaneously perceive local and global features, thereby improving the reconstruction quality of image details. Furthermore, the discriminator enhances the model's ability to capture important features by introducing a self-attention mechanism, and ensures high-quality generated images through a novel perceptual gradient loss function. Simultaneously, a novel Auto-CWT image conversion method is proposed, specifically addressing the problem of weakened regions of interest in fused images in the monitoring field, effectively enhancing the discriminability of target regions.

[0069] Example 2

[0070] Reference Figures 5-7 This is the second embodiment of the present invention. This embodiment is based on the previous embodiment. The difference between this embodiment and embodiment 1 is that it verifies the technical effects of the present invention through simulation verification.

[0071] Specifically, the performance of the SA-DAEDCGAN model in this invention was evaluated through various experiments.

[0072] The mainstream image fusion models selected for comparison are CycleGAN, ResNet, SSRNET, and InfoGAN, using the self-built InsectP dataset. Performance comparison experiments were conducted to evaluate the quality of the images generated by the models. Specifically, the four image fusion algorithms (CycleGAN, ResNet, SSRNET, and InfoGAN) were compared, and all were trained using the same dataset (InsectP) and unchanged original parameter weights. All models were quantitatively evaluated using metrics such as SSIM, PSNR, MSE, and IS, and the results were analyzed. Loss function ablation experiments were also conducted to evaluate the contribution of the proposed loss function to the image fusion task.

[0073] Next, accuracy experiments were conducted on the generated images and the original dataset to evaluate the accuracy of the fused features. Specifically, the recognition accuracy of the fused images on the InsectP dataset was verified using the GoogLeNet model. Five-fold cross-validation was employed, with both the fused image dataset and the original dataset randomly divided into five parts, four for training and one for testing. The convergence loss of the model was observed through 100 iterations, and the recognition accuracy of the two sets of data was compared and analyzed to evaluate the impact of image fusion on recognition quality and accuracy.

[0074] Finally, a model complexity experiment was conducted to evaluate the model's ability to capture features.

[0075] Performance comparison experiments were conducted to compare the image fusion capabilities of five models, including the model in this study, on the self-built InsectP dataset. The performance of the SA-DAEDCGAN model was objectively evaluated using four evaluation metrics, as shown in Table 1.

[0076] Table 1 Quantitative evaluation of the InsectP dataset results

[0077]

[0078] The four evaluation metrics, including the Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), Mean Squared Error (MSE), and Inception Score (IS), are calculated using the following formulas:

[0079]

[0080] Where x and y represent two images to be compared, μ represents the mean of the images, and σ 2 Let C1 represent the variance of the image, σ represent the covariance of the image, and C2 and C1 are constants.

[0081]

[0082] IS = e x (E[D KL (p(x|G(z))||p(x))])

[0083] Among them, e x Let p(x|G(z)) represent the exponential distribution of the KL divergence, p(x|G(z)) represent the probability distribution of the network prediction that G(z) belongs to x, p(x) represent the average class distribution of all generated images, KL represents the Kullback-Leibler divergence, and MSE is the mean squared error.

[0084] As shown in Table 1, the SA-DAEDCGAN model performs excellently in all evaluations, especially in image quality and fidelity, where its performance significantly surpasses other models. SSRNET and CycleGAN rank second and third respectively, although they are close to SA-DAEDCGAN in some metrics, their overall performance is still lacking. In contrast, the lightweight InfoGAN model, due to its architecture optimization to reduce computational resource requirements, performs weaker in these metrics, particularly in detail preservation and visual quality. However, it is worth noting that InfoGAN performs relatively strongly in the IS metric, indicating that the generated images have high diversity. This differs from the design goal of the SA-DAEDCGAN model, which focuses more on extracting and fusing features from regions of strong interest, thus having an advantage in improving the quality of fused images, but slightly inferior to InfoGAN in image diversity. Figure 5 The presentation showcases SSIM heatmaps and MSE difference maps of images fused using various algorithms. The SSIM and MSE maps clearly demonstrate that SA-DAEDCGAN exhibits the smallest residuals in both the error heatmap and difference map. Despite the complex scenes and numerous interference factors in the InsectP dataset, and the similar fusion quality of mainstream algorithms, SA-DAEDCGAN still performs best on this dataset according to quantitative evaluation metrics such as SSIM, PSNR, MSE, and IS. Overall, SA-DAEDCGAN demonstrates a more comprehensive advantage in both overall performance and image fusion quality.

[0085] The loss function ablation experiment combined the proposed perceptual gradient loss with the original content and adversarial loss for training, and evaluated the fused image using the four evaluation metrics mentioned above. The results are shown in Table 2.

[0086] Table 2 Evaluation of fusion results with different loss functions

[0087]

[0088] As shown in Table 2, the adversarial loss slightly improved the fusion accuracy after incorporating the perceptual spectral gradient loss, indicating that this constraint can improve the fusion quality. Furthermore, combining the content loss and adversarial loss with the perceptual gradient loss yielded a more significant improvement, demonstrating the more prominent effect of the perceptual gradient constraint. This indicates that introducing the perceptual gradient loss function can effectively improve the quality of image fusion, with each loss component contributing to the improvement.

[0089] Accuracy experiments were conducted by comparing a dataset created using the images fused and generated according to this invention with the aforementioned RGB image dataset. The GoogLeNet network was used to evaluate both the model's convergence speed and the accuracy of the test set recognition. Figure 7 (a) shows the convergence and loss graphs of the network model trained on the two datasets. Figure 7 (b) is the confusion matrix validated on the test set after the network model was trained. Figure 7 As can be seen from the convergence loss curves of different datasets, the GoogLeNet network trained on the fused dataset converges faster than the network trained on the original dataset, and the former's accuracy of 98.6% on the validation set is also better than the latter's 94.4%. In summary, this indicates that the fused dataset has distinct features, the categories in the dataset are more discriminative, and the image features can be better extracted and learned by the model, thus accelerating the model's convergence.

[0090] The model complexity experiment compares the model with mainstream models to evaluate memory usage and testing time, thus assessing model complexity. The experiment uses 256×256 test images, recording the size (number of parameters) and testing time for each model. The results are shown in Table 3.

[0091] Table 3. Results of Model Complexity Analysis

[0092]

[0093]

[0094] Model size refers to the size of the model. As shown in Table 3, the SA-DAEDCGAN model is larger and takes longer to test compared to the other four models. This is because the model introduces multiple encoder and attention mechanism modules, which increases the number of parameters that need to be processed during testing. This result indicates that the model in this study can capture more image details, thereby improving the quality of the generated images.

[0095] from Figure 5As can be seen, the SA-DAEDCGAN model performs excellently in all evaluations, especially in image quality and fidelity, where its performance significantly outperforms other models. SSRNET and CycleGAN rank second and third respectively, although they are close to SA-DAEDCGAN in some metrics, their overall performance is still lacking. In contrast, the lightweight InfoGAN model, due to its architecture optimization to reduce computational resource requirements, performs weaker in these metrics, particularly in detail preservation and visual quality. However, it is worth noting that InfoGAN performs relatively strongly in the IS metric, indicating that the images it generates have high diversity. This differs from the design goal of the SA-DAEDCGAN model, which focuses more on extracting and fusing features from regions of strong interest, thus having an advantage in improving the quality of fused images, but slightly inferior to InfoGAN in image diversity.

[0096] from Figure 6 As can be seen, the error heatmap and difference map of SA-DAEDCGAN show the smallest residuals. Although the InsectP dataset has complex scenes and numerous interference factors, and the fusion quality of mainstream algorithms is similar, SA-DAEDCGAN still performs best on this dataset according to quantitative evaluation metrics such as SSIM, PSNR, MSE, and IS. Overall, SA-DAEDCGAN demonstrates a more comprehensive advantage in overall performance and image fusion quality.

[0097] Experimental results show that SA-DAEDCGAN outperforms current mainstream algorithms in terms of image quality fusion, feature preservation, and feature enhancement. It can provide higher quality image support for the accurate monitoring of crop diseases and pests, and promote the accurate prediction and monitoring of disease development trends.

[0098] Example 3

[0099] Reference Figure 8 and Figure 9 This is the third embodiment of the present invention. This embodiment is based on the previous embodiment. The difference between this embodiment and embodiment 2 is that, in order to verify the universality of the present invention, a public dataset is used, and fused data and related descriptions are provided.

[0100] Table 4 shows the quantitative evaluation data of the model's results on the public Pavia dataset.

[0101] Quantitative evaluation of Pavia dataset results

[0102]

[0103]

[0104] As shown in Table 4, the SA-DAEDCGAN model described in this invention demonstrates significant advantages across all key metrics. Particularly in terms of image structural similarity and fidelity, it proves the model's superior performance in noise suppression and reconstruction error reduction. Furthermore, the model remains applicable when processing complex public Pavia datasets.

[0105] This embodiment compares four image fusion algorithms (CycleGAN, ResNet, SSRNET, and InfoGAN) and trains them using the same dataset (Pavia) and unchanged original parameter weights. All models are quantitatively evaluated using metrics such as SSIM, PSNR, MSE, and IS, and the results are analyzed.

[0106] from Figure 8 As can be seen, the SA-DAEDCGAN model demonstrates significant advantages across all key metrics, particularly in image structural similarity and fidelity, where it excels with an SSIM of 0.9121 and a PSNR of 39.3256. This indicates that SA-DAEDCGAN can effectively handle complex spectral information and generate high-quality fused images while preserving image details, far surpassing other models. Furthermore, SA-DAEDCGAN also exhibits the lowest MSE value of 23.6594, significantly outperforming CycleGAN (35.3698) and SRNET (41.3652), further demonstrating its advantages in noise suppression and reconstruction error. When processing the complex Pavia dataset, the low MSE value indicates that SA-DAEDCGAN can better preserve the high-dimensional information of the original image, avoiding excessive distortion or noise during the fusion process.

[0107] from Figure 9 As can be seen, SA-DAEDCGAN exhibits the smallest bias, and the MSE difference map further validates its superior pixel-level error performance. In contrast, mainstream algorithms such as CycleGAN, ResNet, and SSRNET demonstrate similar fusion quality, while InfoGAN performs the worst. Overall, the SA-DAEDCGAN model excels in its superior image quality, fidelity, and noise suppression capabilities, particularly demonstrating outstanding performance on complex datasets.

[0108] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the technical solutions of the present invention.

Claims

1. A hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint sensing gradient loss, characterized in that: Includes the following steps, Collect RGB and hyperspectral images, transfer the collected images to a computer, preprocess the images, summarize the preprocessed data, and generate an image fusion dataset. The image fusion dataset is input into the SA-DAEDCGAN model. Through the primary processing module and the generator in the SA-DAEDCGAN model, pseudo-images are generated via convolution, encoding, and decoding. These pseudo-images are then fed into the discriminator in the SA-DAEDCGAN model. The discriminator analyzes the differences between the pseudo-images and the real images. The pseudo-images and real images are compared using the total perceptual gradient loss function L(G,D) to calculate the total perceptual gradient loss, assessing their similarity in the feature space. This loss is then fed back to the generator to optimize the generation process. If the discriminator determines the image is real, it outputs the result; if it determines it is fake, it feeds back to the generator for continued training. The formula for the total perceived gradient loss function is L(G,D)=αL GAN +βL Con +λL PG ; Among them, L GAN L represents the adversarial loss function; Con L represents the image content loss function; PG Let represent the sub-sensory gradient loss function; α, β, and λ represent learnable adaptive weight coefficients. The formula for the adversarial loss function is as follows: Where x represents the real image x, and G(z) represents the image generated by the generator. For the true image distribution P data The expected value of P on (x), data (x) represents the probability distribution of the real image. To generate an image in P data The expected value on (z), P data (z) represents the probability distribution of the image generated by the generator, D(x) represents the probability output of the discriminator that the input image is a real image, and D(G(z)) represents the probability output of the discriminator that the image generated by the generator is a real image. The formula for the image content loss function is as follows: Where H and W represent the height and width of the image, i is the row index of a pixel in the image, j is the column index of a pixel in the image, and x... ij Let G(z) be the pixel value of the real image at coordinates (i, j). ij To generate the pixel value of image G(z) at coordinates (i, j); The formula for the sub-sensory gradient loss function is: L PG =λ1L P +λ2L G Among them, L P L represents perceived loss. G Represents gradient loss, These are represented as horizontal and vertical gradient operators, respectively.

2. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint sensing gradient loss as described in claim 1, characterized in that: The generator introduces a denoising autoencoder network architecture in the input layer to encode and denoise the input hyperspectral and RGB images, extract and integrate features, and inject them into the new image to achieve fusion.

3. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint sensing gradient loss as described in claim 2, characterized in that: The discriminator incorporates a self-attention mechanism into its structure.

4. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint sensing gradient loss as described in any one of claims 1 to 3, characterized in that: During preprocessing, for hyperspectral images, the one-dimensional average reflectance curve of the region of interest is first extracted using ENVI. Then, the one-dimensional feature curve data is transformed into two-dimensional feature image data using the Auto-CWT method, ultimately forming an irrelevant feature dataset. The Auto-CWT method incorporates an autoencoder into the continuous wavelet transform method, and the formula is as follows. z=f encoder (CWT(x(t))) W x (a,b)=f decoder (z) Where z represents the low-dimensional space, f encoder The mapping function f of the encoder. decoder The decoder's mapping function is represented by CWT(x(t)), which represents the original CWT feature matrix, and W... x (a,b) represents the reconstruction matrix, and L represents the loss function.

5. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint sensing gradient loss as described in any one of claims 1 to 3, characterized in that: The primary processing module first upsamples the RGB image to obtain image information with the same spatial size as the hyperspectral image, and then extracts shallow information from the RGB and hyperspectral images through two convolutional layers and maps them to the same spectral dimension.

Images