Hyperspectral and RGB image fusion method based on SA-DAEDCGAN combined perception gradient loss

Abstract

The invention discloses a hyperspectral and RGB image fusion method based on SA-DAEDCGAN combined perception gradient loss, and the method comprises the following steps: collecting an RGB image and a hyperspectral image, transmitting the collected images to a computer for preprocessing, summarizing the preprocessed data, and generating an image fusion data set; a data set is input into the model, a pseudo image is generated through a primary processing module and a generator in the model through convolution, coding and decoding, the pseudo image is transmitted into a discriminator in the model, the discriminator analyzes the difference between the pseudo image and a real image, the pseudo image and the real image calculate the total perceptual gradient loss through a total perceptual gradient loss function, and the final image is obtained. The similarity of the two images in the feature space is evaluated, the obtained loss is fed back to the generator, the generation process is optimized, and if the discriminator judges that the image is true, a result is output; if so, feeding back to the generator, and continuing training; by using the method of the invention, RGB and hyperspectral images can be fused completely, accurately and rapidly.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss. Background Art

[0002] Crop pests and diseases have always been an important factor restricting agricultural production due to their many types, wide impact range and local outbreaks. They can have a great impact on crop yield and quality. In recent years, the occurrence and prevalence of pests and diseases have been increasing, and the prevention and control of pests and diseases has become more arduous. According to the Food and Agriculture Organization of the United Nations (FAO), the annual grain production loss caused by pests and diseases in the world is about 1 / 4 of the total output, of which the loss caused by diseases is 14% and the loss caused by pests is 10%. Therefore, accurately and timely obtaining the spatial distribution and development trend of pests and diseases in farmland is the key to disease prevention and control. With the development of image processing technology and the popularity of RGB cameras, people can easily obtain RGB images, and the effective information captured by RGB images has played an important role in the field of crop pest and disease detection. However, RGB images mainly contain information in the three channels of red, green and blue. Although they can replace manual inspection and accurately provide information on pest and disease erosion, they are powerless for more detailed spectral feature information of pests and diseases. Therefore, hyperspectral images came into being. Hyperspectral images use hundreds or thousands of continuous narrow bands to express specific scene information. This information not only contains rich spatial information, but also has spectral information that is not available in RGB images, which can more accurately monitor crop pest and disease information. However, hyperspectral images have limited spectral information representation, and sometimes it is difficult to fully express all features. In addition, the use of spectral or spatial scanning methods to capture three-dimensional data cubes often requires a lot of exposure time. The imaging method that sacrifices temporal resolution to obtain high-quality spectral resolution also hinders the application of hyperspectral images in the field of crop pest and disease monitoring. Therefore, there is an urgent need for a method that can completely, accurately and quickly fuse RGB and hyperspectral images. Summary of the invention

[0003] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0004] In view of the above problems and / or the problems existing in the existing image processing technology, the present invention is proposed.

[0005] Therefore, the problem to be solved by the present invention is that the fusion speed of RGB and hyperspectral images in the prior art is slow and cannot fully express all features. The present invention can completely, accurately and quickly fuse RGB and hyperspectral images.

[0006] To solve the above technical problems, the present invention provides the following technical solution: a hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss, which comprises the following steps:

[0007] Collect RGB images and hyperspectral images, transfer the collected images to a computer, preprocess the images, summarize the preprocessed data, and generate an image fusion data set;

[0008] The image fusion dataset is input into the SA-DAEDCGAN model, and a pseudo image is generated through convolution, encoding and decoding by the primary processing module and the generator in the SA-DAEDCGAN model. The pseudo image is passed into the discriminator in the SA-DAEDCGAN model, and the discriminator analyzes the difference between it and the real image. The pseudo image and the real image are passed through the total perceptual gradient loss function L(G,D), and the total perceptual gradient loss is calculated to evaluate the similarity between the two in the feature space. The obtained loss is fed back to the generator to optimize the generation process. If the discriminator determines that the image is real, the result is output; if it is determined to be forged, it is fed back to the generator to continue training.

[0009] As a preferred solution of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in the present invention, the generator introduces a denoising autoencoder network architecture at the input layer, encodes and denoises the input hyperspectral and RGB images, extracts and integrates features, and injects them into a new image to achieve fusion.

[0010] As a preferred solution of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in the present invention, wherein: the discriminator incorporates a self-attention mechanism into the structure.

[0011] As a preferred solution of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in the present invention, during preprocessing, for the hyperspectral image, the one-dimensional average reflectance curve of the area of ​​interest is first extracted by ENVI, and then the one-dimensional feature curve data is converted into two-dimensional feature image data by the Auto-CWT method, finally forming an irrelevant feature data set.

[0012] As a preferred solution of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in the present invention, the Auto-CWT method incorporates an automatic encoder into the continuous wavelet transform method, and the formula is as follows:

[0013] z=f encoder (CWT(x(t)))

[0014] W x (a,b)=f decoder (z)

[0015]

[0016] Among them, z represents the low-dimensional space, f encoder represents the mapping function of the encoder, f decoder represents the mapping function of the decoder, CWT(x(t)) represents the original CWT feature matrix, W x (a, b) represents the reconstruction matrix, and L represents the loss function.

[0017] As a preferred solution of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in the present invention, the primary processing module first upsamples the RGB image to obtain image information of the same spatial size as the hyperspectral image, and then extracts shallow information in RGB and the hyperspectral image through two convolutional layers, and maps them to the same spectral dimension.

[0018] As a preferred solution of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in the present invention, the formula of the total perceptual gradient loss function is: L(G,D)=αL GAN +βL Con +λL PG .

[0019] Among them, L GAN represents the adversarial loss function; L Con represents the image content loss function; L PG represents the perceptual gradient loss function; α, β, λ represent learnable adaptive weight coefficients.

[0020] As a preferred solution of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in the present invention, the formula of the adversarial loss function is:

[0021]

[0022] Among them, x represents the real image x, G(z) represents the image generated by the generator, and Ex～pdata(x) is the real image distribution P data The expected value on (x), P data (x) is the probability distribution of the real image, To generate an image in P data The expected value on (z), P data (z) is the probability distribution of the image generated by the generator, D(x) is the probability output of the discriminator that the input image is a real image, and D(G(z)) is the probability output of the discriminator that the image generated by the generator is a real image.

[0023] As a preferred solution of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in the present invention, the image content loss function measures the similarity between the generated image and the target image by calculating the difference between the high-level semantic features of the two. The formula is:

[0024]

[0025] Where H and W represent the height and width of the image, i is the row index of the pixel in the image, j is the column index of the pixel in the image, and x is the ij is the pixel value of the real image at coordinate (i, j), G(z) ij To generate the pixel value of the image G(z) at the coordinate (i, j).

[0026] As a preferred solution of the hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss in the present invention, the formula of the perceptual gradient loss function is:

[0027] L PG =λ1L P +λ2L G

[0028]

[0029] Among them, L P represents the perceptual loss, L G represents the gradient loss, Represented as horizontal and vertical gradient operators respectively.

[0030] The beneficial effects of the present invention are as follows: the present invention combines the advantages of generative adversarial networks, self-attention mechanisms and denoising autoencoders, proposes a SA-DAEDCGAN model that fuses hyperspectral images and RGB images, introduces a primary processing module and a denoising autoencoder architecture into the generator, so that the network can perceive local and global features at the same time; constructs a discriminator with a self-attention mechanism to ensure the quality of the reconstructed image; proposes a new perceptual gradient loss function to improve the training stability of the model and generate higher quality and more realistic images. In addition, a new Auto-CWT image conversion method is proposed to solve the problem of weakening the feature area of ​​interest in fused images in some monitoring fields, thereby improving the accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0032] Figure 1 It is a schematic diagram of the process of the present invention.

[0033] Figure 2 This is the Auto-CWT preprocessing process in the present invention.

[0034] Figure 3 A schematic diagram of the generator in the model.

[0035] Figure 4 Schematic diagram of the discriminator in the model.

[0036] Figure 5 1 is a comparison diagram of the evaluation results of the performance comparison experiment in Example 2 ((a) is a stacked diagram of the evaluation results, and (b) is a bar graph of the evaluation results).

[0037] Figure 6 This is a comparison heat map of the evaluation results of the performance comparison experiment in Example 2.

[0038] Figure 7 The results of the accuracy experiment in Example 2 are shown in Figure 2 (Figure (a) is a model training accuracy curve graph, Figure (b) is a model training loss curve graph, Figure (c) is a confusion matrix heat map of the fusion data set validation set results, and Figure (d) is a confusion matrix heat map of the original data set validation set results).

[0039] Figure 8 1 is a comparison diagram of the evaluation results of the performance comparison experiment in Example 3 (Figure (a) is a stacked diagram of the evaluation results, and Figure (b) is a bar graph of the evaluation results).

[0040] Fig. 9 This is a thermal diagram comparing the evaluation results of the performance comparison experiment in Example 3. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0042] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0043] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0044] Example 1

[0045] Reference Figure 1 to Figure 4 , which is the first embodiment of the present invention, provides a hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss, which can completely, accurately and quickly fuse RGB and hyperspectral images.

[0046] The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss includes the following steps:

[0047] Step 1, data acquisition and preprocessing: collect RGB images and hyperspectral images, transfer the collected images to the computer, and preprocess the images. During preprocessing, for the hyperspectral image, first extract the one-dimensional average reflectance curve of the area of ​​interest through ENVI, and then use the Auto-CWT method to convert the one-dimensional feature curve data into two-dimensional feature image data, and finally form an irrelevant feature data set, summarize the preprocessed data, and generate an image fusion data set;

[0048] The Auto-CWT method incorporates an autoencoder into the continuous wavelet transform method. The formula is as follows:

[0049] z=f encoder (CWT(x(t)))

[0050] W x (a,b)=f decoder (z)

[0051]

[0052] Among them, z represents the low-dimensional space, f encoder represents the mapping function of the encoder, f decoder represents the mapping function of the decoder, CWT(x(t)) represents the original CWT feature matrix, W x (a, b) represents the reconstruction matrix, L represents the loss function;

[0053] Step 2, model construction and optimization: The image fusion dataset is input into the SA-DAEDCGAN model, and a pseudo image is generated through convolution, encoding and decoding by the primary processing module and the generator in the SA-DAEDCGAN model. The pseudo image is passed to the discriminator in the SA-DAEDCGAN model, and the discriminator analyzes the difference between it and the real image. The pseudo image and the real image pass through the total perceptual gradient loss function L(G, D), calculate the total perceptual gradient loss, evaluate the similarity between the two in the feature space, and feed the obtained loss back to the generator to optimize the generation process. If the discriminator determines that the image is real, the result is output; if it is determined to be forged, it is fed back to the generator to continue training.

[0054] Among them, the primary processing module first upsamples the RGB image to obtain image information of the same spatial size as the hyperspectral spectrum, and then extracts the shallow information in RGB and hyperspectral spectrum through two convolutional layers and maps them to the same spectral dimension.

[0055] Generator, introduces a denoising autoencoder network architecture at the input layer, encodes and de-noises the input hyperspectral and RGB images, extracts and integrates features, and injects them into the new image to achieve fusion, such as Figure 3 As shown in the figure, Conv1~Conv4 are all convolutional layers; the denoising autoencoder network architecture is introduced between Conv3 and Conv4; the main feature processing is to extract shallow information in RGB and hyperspectral through two convolutional layers and map them to the same spectral dimension; the discriminator incorporates the self-attention mechanism into the structure, and combines the self-attention mechanism with the convolutional layer to enhance the ability to identify the authenticity of the image, such as Figure 4As shown, the image passes through three convolutional layers (Conv1, Conv2, Conv3) in sequence. These convolutional layers are used for feature extraction and capture the complex spatial features in the image step by step. After the convolutional layer, the architecture contains a self-attention mechanism, which is used in conjunction with an additional convolutional layer (Conv4). The self-attention mechanism can help the network focus on important parts of the image and capture long-distance dependencies between image pixels, thereby selecting more relevant features to make judgments based on the weights of different areas of the image. After the self-attention layer, the output passes through a softmax layer (normalized exponential function layer) to produce a probability distribution to determine whether the input image is more likely to be real or generated; ultimately, the output of the discriminator is a binary classification result: the input image is classified as fake or real.

[0056] The formula of the total perceptual gradient loss function is: L(G,D) = αL GAN +βL Con +λL PG .

[0057] Among them, L GAN represents the adversarial loss function; L Con represents the image content loss function; L PG represents the perceptual gradient loss function; α, β, λ represent learnable adaptive weight coefficients.

[0058] Adversarial loss function L GAN , by minimizing the loss of the model on adversarial samples, the robustness of the model to input perturbations is improved. The formula is:

[0059]

[0060] Among them, x represents the real image x, G(z) represents the image generated by the generator, is the real image distribution P data The expected value on (x), P data (x) is the probability distribution of the real image, To generate an image in P data The expected value on (z), P data (z) is the probability distribution of the image generated by the generator, D(x) is the probability output of the discriminator that the input image is a real image, and D(G(z)) is the probability output of the discriminator that the image generated by the generator is a real image.

[0061] Image content loss function L Con , by calculating the difference between the high-level semantic features of the two, the similarity between the generated image and the target image is measured. The formula is:

[0062]

[0063] Where H and W represent the height and width of the image, i is the row index of the pixel in the image, j is the column index of the pixel in the image, and x is the ij is the pixel value of the real image at coordinate (i, j), G(z) ij To generate the pixel value of the image G(z) at the coordinate (i, j).

[0064] Perceptual gradient loss function L PG , weighted summation of perceptual loss and gradient loss can achieve the purpose of simultaneously considering the performance of the generated image in feature space and gradient edge information, thereby improving the visual quality and detail retention of the image. The formula is:

[0065] L PG =λ1L P +λ2L G

[0066]

[0067] Among them, L P represents the perceptual loss, L G represents the gradient loss, Represented as horizontal and vertical gradient operators respectively.

[0068] This application significantly improves the fusion effect of RGB images and hyperspectral images by introducing the SA-DAEDCGAN model. This model innovatively combines generative adversarial networks, self-attention mechanisms, and denoising autoencoders to solve the technical bottlenecks of limited spectral information of single RGB images and low resolution of hyperspectral images, and overcomes the problems of existing fusion algorithms in losing important features and generating low-quality images. Specifically, the primary processing module and denoising autoencoder architecture added to the generator enable the network to perceive local and global features at the same time, thereby improving the reconstruction quality of image details. In addition, the discriminator enhances the model's ability to capture important features by introducing a self-attention mechanism, and ensures the high quality of generated images by designing a new perceptual gradient loss function. At the same time, a new Auto-CWT image conversion method is proposed, which is specifically aimed at the problem of weakening the feature area of ​​interest in the fused image in the monitoring field, and effectively enhances the recognizability of the target area.

[0069] Example 2

[0070] Reference Figure 5 to Figure 7 , which is the second embodiment of the present invention. This embodiment is based on the previous embodiment. The difference between this embodiment and embodiment 1 is that it verifies the technical effect of the present invention through the technical means of simulation verification.

[0071] Specifically, the performance of the SA-DAEDCGAN model in the present invention is evaluated through various experiments.

[0072] The mainstream image fusion comparison models selected are CycleGAN, ResNet, SSRNET and InfoGAN, and the dataset is the self-built InsectP dataset. The quality of the images generated by the model is evaluated through performance comparison experiments. Specifically, four image fusion algorithms (CycleGAN, ResNet, SSRNET, InfoGAN) are compared, and the same dataset (InsectP) and the original parameter weights are used for training. All models are quantitatively evaluated and analyzed through indicators such as SSIM, PSNR, MSE and IS; and loss function ablation experiments are performed to evaluate the contribution of the proposed loss function to the image fusion task.

[0073] Next, we conducted an accuracy experiment on the generated images and the original dataset to evaluate the accuracy of the fused features. Specifically, we used the GoogLeNet model to verify the recognition accuracy of the fused images of the InsectP dataset. Using a five-fold cross validation, we randomly divided the fused image dataset and the original dataset into five parts, four of which were used for training and the remaining one for testing. We observed the convergence loss of the model through 100 iterations, and compared the recognition accuracy of the two sets of data to evaluate the impact of image fusion on recognition quality and accuracy.

[0074] Finally, a model complexity experiment is conducted to evaluate the model's ability to capture features.

[0075] The performance comparison experiment compares the image fusion capabilities of five models including the model in this study on the self-built InsectP dataset, and uses four evaluation indicators to objectively evaluate the performance of the SA-DAEDCGAN model, as shown in Table 1.

[0076] Table 1 Quantitative evaluation of the results of the InsectP dataset

[0077]

[0078] Four evaluation indicators, including structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), mean square error (MSE) and Inception Score (IS), are calculated as follows:

[0079]

[0080] Among them, x and y represent two images to be compared, μ represents the mean of the images, and σ 2 represents the variance of the image, σ represents the covariance of the image, and C1 and C2 are constants.

[0081]

[0082] IS=e x (E[D KL (p(x|G(z))||p(x))])

[0083] Among them, e x represents the exponential of the KL divergence, p(x|G(z)) represents the probability distribution of the network predicting that G(z) belongs to x, p(x) represents the average category distribution of all generated images, KL represents the Kullback-Leibler divergence, and MSE is the mean square error.

[0084] As can be seen from Table 1, the SA-DAEDCGAN model performs well in all evaluations, especially in image quality and fidelity, where its performance is significantly better than other models. Meanwhile, SSRNET and CycleGAN rank second and third respectively, and although they are close to SA-DAEDCGAN in some indicators, they are still lacking in overall performance. In contrast, the lightweight InfoGAN model performs weaker in these indicators, especially in detail preservation and visual quality, due to its architecture optimization to reduce the demand for computing resources. However, it is worth noting that InfoGAN performs relatively strongly in the IS indicator, indicating that the images it generates have a high diversity. This is different from the design goal of the SA-DAEDCGAN model, which focuses more on extracting and fusing features of strong regions of interest, so it has more advantages in improving the quality of fused images, but is slightly inferior to InfoGAN in image diversity. Figure 5 The SSIM heatmaps and MSE difference maps of the fused images using various algorithms are shown. The SSIM and MSE plots clearly show that the error heatmap and difference map of SA-DAEDCGAN show the smallest residual. Although the InsectP dataset has complex scenes and many interference factors, and the fusion quality of mainstream algorithms is similar, SA-DAEDCGAN still performs best on this dataset through quantitative evaluation indicators such as SSIM, PSNR, MSE, and IS. In general, SA-DAEDCGAN shows more comprehensive advantages in overall performance and image fusion quality.

[0085] In the loss function ablation experiment, the proposed perceptual gradient loss is combined with the original content and adversarial loss for training, and their fused images are evaluated using the four evaluation indicators mentioned above. The results are shown in Table 2.

[0086] Table 2 Evaluation of fusion results of different loss functions

[0087]

[0088] As can be seen from Table 2, after adding the perceptual spectral gradient loss, the adversarial loss slightly improves the fusion accuracy, indicating that this constraint can improve the fusion quality. Furthermore, when the content loss and adversarial loss are combined with the perceptual gradient loss, the improvement effect is more significant, indicating that the effect of the perceptual gradient constraint is more prominent. This shows that the introduction of the perceptual gradient loss function can effectively improve the quality of image fusion, and each loss component contributes to improving the fusion quality.

[0089] In the accuracy experiment, the images generated by the fusion of the present invention are used to make a data set and compared with the RGB image data set, and the GoogLeNet network is used to evaluate the model convergence speed or the test set recognition accuracy. Figure 7 (a) is the convergence and loss diagram of the network model trained on two data sets. Figure 7 (b) is the confusion matrix of the network model after training and verification on the test set. Figure 7 It can be seen from the convergence loss curves of different data sets that the GoogLeNet network using the fusion data set converges faster than the network trained using the original data set, and the accuracy of the former on the validation set is 98.6%, which is better than the latter's accuracy of 94.4%. In summary, the fusion data set has obvious characteristics, the categories in the data set have obvious distinctions, and the features of the image can be better extracted and learned by the model, thereby accelerating the convergence of the model.

[0090] The model complexity experiment evaluates the model complexity by comparing the model with the mainstream models, evaluating the memory size and test time. The experiment uses a 256×256 test image and records the volume (number of parameters) and test time of each model. The results are shown in Table 3.

[0091] Table 3 Model complexity analysis results

[0092]

[0093]

[0094] Model size is the model size. As can be seen from Table 3, the SA-DAEDCGAN model is larger and takes longer to test than the other four models. This is because the model introduces multiple encoders and attention mechanism modules, which increases the number of parameters that need to be processed during testing. This result shows that the model in this study can capture more image details, thereby improving the quality of generated images.

[0095] from Figure 5As can be seen from the table, the SA-DAEDCGAN model performs well in all evaluations, especially in image quality and fidelity, where its performance is significantly better than other models. At the same time, SSRNET and CycleGAN rank second and third respectively. Although they are close to SA-DAEDCGAN in some indicators, they are still lacking in overall performance. In contrast, the lightweight InfoGAN model performs weaker in these indicators, especially in detail preservation and visual quality, due to its optimized architecture to reduce the demand for computing resources. However, it is worth noting that InfoGAN performs relatively strongly in the IS indicator, indicating that the images it generates have a high diversity. This is different from the design goal of the SA-DAEDCGAN model, which focuses more on extracting and fusing features of strong regions of interest, so it has more advantages in improving the quality of fused images, but is slightly inferior to InfoGAN in image diversity.

[0096] from Figure 6 As can be seen in the figure, the error heatmap and difference map of SA-DAEDCGAN show the smallest residual. Although the InsectP dataset has complex scenes and many interference factors, and the fusion quality of mainstream algorithms is similar, SA-DAEDCGAN still performs best on this dataset through quantitative evaluation indicators such as SSIM, PSNR, MSE, and IS. In general, SA-DAEDCGAN shows more comprehensive advantages in overall performance and image fusion quality.

[0097] Experimental results show that SA-DAEDCGAN is superior to current mainstream algorithms in terms of fusion image quality, feature preservation and feature enhancement. It can provide higher quality image support for precise monitoring of crop diseases and pests, and promote accurate prediction and monitoring of disease development trends.

[0098] Example 3

[0099] Reference Figure 8 and Fig. 9 , which is the third embodiment of the present invention. This embodiment is based on the previous embodiment. The difference between this embodiment and embodiment 2 is that, in order to verify the universality of the present invention, a public data set is used, and fusion data and related instructions are provided.

[0100] Table 4 shows the quantitative evaluation data of the model on the public Pavia dataset.

[0101] Quantitative evaluation of results on the Pavia dataset

[0102]

[0103]

[0104] As can be seen from Table 4, the SA-DAEDCGAN model of the present invention shows significant advantages in all key indicators. In particular, in terms of structural similarity and fidelity of images, the model's advantages in noise suppression and reconstruction error are demonstrated, and the model of the present invention is still applicable when processing the complex public Pavia dataset.

[0105] This example compares four image fusion algorithms (CycleGAN, ResNet, SSRNET, InfoGAN), and uses the same dataset (Pavia) and the original parameter weights to train. All models are quantitatively evaluated using indicators such as SSIM, PSNR, MSE, and IS, and the results are analyzed.

[0106] from Figure 8 As can be seen from the figure, the SA-DAEDCGAN model shows significant advantages in all key indicators, especially in terms of structural similarity and fidelity of images, where SA-DAEDCGAN performs well with SSIM 0.9121 and PSNR 39.3256. This shows that SA-DAEDCGAN can effectively process complex spectral information and generate high-quality fused images while maintaining image details, which is far superior to other models. In addition, SA-DAEDCGAN also shows the lowest value of 23.6594 in the MSE indicator, which is significantly better than CycleGAN35.3698 and SRNET41.3652, which further proves the advantages of the model in noise suppression and reconstruction error. When processing the complex Pavia dataset, the low MSE value shows that SA-DAEDCGAN can better maintain the high-dimensional information of the original image and avoid excessive distortion or noise during the fusion process.

[0107] from Fig. 9 As can be seen from the figure, SA-DAEDCGAN has the smallest deviation, and the MSE difference map further verifies its excellent pixel-level error performance. In contrast, mainstream algorithms such as CycleGAN, ResNet, and SSRNET show similar fusion quality, while InfoGAN performs the worst. In general, the advantage of the SA-DAEDCGAN model lies in its excellent image quality, fidelity, and noise suppression capabilities, especially on complex datasets.

[0108] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the technical solutions of the present invention.

Claims

1. A hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss, characterized by: The following steps are included: Collect RGB images and hyperspectral images, transfer the collected images to a computer, preprocess the images, summarize the preprocessed data, and generate an image fusion data set; The image fusion dataset is input into the SA-DAEDCGAN model, and a pseudo image is generated through convolution, encoding and decoding by the primary processing module and the generator in the SA-DAEDCGAN model. The pseudo image is passed into the discriminator in the SA-DAEDCGAN model, and the discriminator analyzes the difference between it and the real image. The pseudo image and the real image are passed through the total perceptual gradient loss function L(G,D), and the total perceptual gradient loss is calculated to evaluate the similarity between the two in the feature space. The obtained loss is fed back to the generator to optimize the generation process. If the discriminator determines that the image is real, the result is output; if it is determined to be forged, it is fed back to the generator to continue training.

2. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss as claimed in claim 1, characterized in that: The generator introduces a denoising autoencoder network architecture at the input layer, encodes and denoises the input hyperspectral and RGB images, extracts and integrates features, and injects them into new images to achieve fusion.

3. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss as claimed in claim 2, characterized in that: The discriminator incorporates a self-attention mechanism into its structure.

4. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss according to any one of claims 1 to 3, characterized in that: During preprocessing, for hyperspectral images, the one-dimensional average reflectance curve of the area of ​​interest is first extracted through ENVI, and then the one-dimensional feature curve data is converted into two-dimensional feature image data through the Auto-CWT method, finally forming an irrelevant feature data set.

5. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss as claimed in claim 4, characterized in that: The Auto-CWT method incorporates an autoencoder into the continuous wavelet transform method. The formula is as follows: z=f encoder (CWT(x(t))) W x (a,b)=f decoder (z) Among them, z represents the low-dimensional space, f encoder represents the mapping function of the encoder, f decoder represents the mapping function of the decoder, CWT(x(t)) represents the original CWT feature matrix, W x (a, b) represents the reconstruction matrix, and L represents the loss function.

6. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss according to any one of claims 1 to 3, characterized in that: The primary processing module first obtains image information of the same spatial size as the hyperspectral spectrum by upsampling the RGB image, and then extracts shallow information in the RGB and hyperspectral spectrum through two convolutional layers and maps them to the same spectral dimension.

7. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss according to any one of claims 1 to 3, characterized in that: The formula of the total perceptual gradient loss function is, L(G,D) = αL GAN +βL Con +λL PG ; Among them, L GAN represents the adversarial loss function; L Con represents the image content loss function; L PG represents the perceptual gradient loss function; α, β, λ represent learnable adaptive weight coefficients.

8. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss as claimed in claim 7, characterized in that: The formula of the adversarial loss function is: Among them, x represents the real image x, G(z) represents the image generated by the generator, is the real image distribution P data The expected value on (x), P data (x) is the probability distribution of the real image, To generate an image in P data The expected value on (z), P data (z) is the probability distribution of the image generated by the generator, D(x) is the probability output of the discriminator that the input image is a real image, and D(G(z)) is the probability output of the discriminator that the image generated by the generator is a real image.

9. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss as claimed in claim 7, characterized in that: The formula of the image content loss function is: Where H and W represent the height and width of the image, i is the row index of the pixel in the image, j is the column index of the pixel in the image, and x is the pixel index of the image. ij is the pixel value of the real image at coordinate (i, j), G(z) ij To generate the pixel value of the image G(z) at the coordinate (i, j).

10. The hyperspectral and RGB image fusion method based on SA-DAEDCGAN joint perceptual gradient loss as claimed in claim 7, characterized in that: The formula of the sub-perception gradient loss function is: L PG =λ1L P +λ2L G Among them, L P represents the perceptual loss, L G represents the gradient loss, Represented as horizontal and vertical gradient operators respectively.

Images