Hyperspectral image super-classification and classification joint optimization method based on diffusion model
By adopting the super-segment and classification joint optimization method based on diffusion model in hyperspectral image processing, the problems of low image quality and poor classification accuracy in the prior art are solved, and efficient joint optimization of image super-segment and classification tasks are achieved, improving the improvement of the comprehensive optimization of the super-segment and classification tasks. The spatial resolution and classification accuracy of the image are obtained.
Patent Information
- Application Number
- CN202410821756.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-24
- Publication Date
- 2024-11-12
AI Technical Summary
The existing hyperspectral image super-segment methods cannot effectively meet the needs of image classification tasks, resulting in insufficient high-quality images generated and affecting classification performance. In addition, traditional methods rely on prior information to be difficult to adapt to specific scenarios, ignoring the needs of image classification tasks.
Using the hyperspectral image super-segment and classification joint optimization method based on diffusion model, the image is realized through collaborative iterative super-segment and classification diffusion model, ring multi-layer optimization network, multi-scale feature fusion mechanism and class-guided super-segment dynamic refinement mechanism Multitask learning of super-scoring and classification tasks, gradually reconstructing high-quality images.
Based on the diffusion model, combined with super-segment and classification networks, the joint optimization of image super-segment and classification tasks is realized, which improves the spatial resolution and classification accuracy of the image, meets the needs of high-precision hyperspectral image processing, and overcomes traditional methods limitations.
Smart Images

Figure CN118941448A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image processing, and in particular relates to a hyperspectral image super-resolution and classification joint optimization method based on a diffusion model. Background Art
[0002] At present, hyperspectral images contain hundreds of continuous narrow spectral bands, which enable the capture of fine spectral features in the observed scene. Therefore, hyperspectral images have been widely used in agriculture, marine protection, forest protection and other fields. However, due to the limitations of hyperspectral imaging equipment, there is a trade-off between spatial resolution, spectral resolution and signal-to-noise ratio. In order to obtain higher spectral resolution, spatial resolution is often sacrificed. In practical applications, low spatial resolution hyperspectral images cannot meet the needs of certain application scenarios. To solve this problem, super-resolution (also known as super-resolution) technology came into being, which can reconstruct high-resolution images from low-resolution images, thereby improving the spatial resolution of hyperspectral images.
[0003] In recent years, a large number of super-resolution methods have been proposed, which can be roughly divided into two categories: traditional super-resolution methods and deep learning-based super-resolution methods. Traditional super-resolution methods mainly include two categories. The first category is based on interpolation methods, which are simple to implement and computationally efficient, but the reconstructed images often cannot restore high-frequency details. The second category is based on reconstruction methods, which mainly use effective spatial or spectral statistical distributions to restore high-frequency details, such as sparse regularization, low rank, and non-local self-similarity. However, these methods require priors to achieve better performance, but it is usually difficult to obtain effective priors that are suitable for specific scenarios.
[0004] Due to the significant advantages of deep learning in feature extraction, super-resolution methods based on deep learning have received widespread attention. Super-resolution methods based on convolutional neural networks (CNNs) directly learn the end-to-end mapping between low-resolution and high-resolution images. Subsequently, super-resolution methods based on generative adversarial networks (GANs) were introduced into the field of image super-resolution, effectively enhancing the ability to restore high-frequency details in images. In addition, transformer-based super-resolution methods significantly improve the fineness and spectral fidelity of image reconstruction by capturing rich local and global multi-scale information. In recent years, super-resolution methods based on diffusion models have shown potential in the field of image super-resolution, which gradually generates finer images through a step-by-step refinement strategy.
[0005] Typically, super-resolution is a preprocessing technique for specific application tasks (e.g., image classification tasks). High-resolution hyperspectral images obtained through super-resolution are crucial for the interpretation of image classification tasks. However, most existing super-resolution methods usually ignore the requirements of image classification tasks. Therefore, these methods cannot effectively generate high-quality images that meet image classification tasks, which affects the classification performance.
[0006] Through the above analysis, the problems and defects of the prior art are as follows:
[0007] (1) In practical applications, low spatial resolution hyperspectral images cannot meet the needs of certain application scenarios.
[0008] (2) Traditional image super-resolution techniques require priors to achieve better performance, but it is usually difficult to obtain effective priors that are suitable for specific scenarios.
[0009] (3) Existing image super-resolution techniques usually ignore the requirements of image classification tasks and cannot effectively generate high-quality images that meet image classification tasks, thus affecting classification performance. Summary of the invention
[0010] In view of the problems existing in the prior art, the present invention provides a hyperspectral image super-resolution and classification joint optimization method based on a diffusion model.
[0011] The present invention is implemented as follows: a hyperspectral image super-resolution and classification joint optimization method based on a diffusion model comprises the following steps:
[0012] Step 1: Input a hyperspectral image with low spatial resolution of the target area and preprocess the image;
[0013] Step 2: Generate a sample set, select a certain number of samples as the training set, and the remaining samples as the test set;
[0014] Step 3: Design a collaborative iterative super-resolution and classification diffusion model. In the forward process, gradually add Gaussian noise to the reference image to degenerate the image into a pure noise state, providing a basis for the subsequent reverse denoising process.
[0015] Step 4: In the reverse denoising process, a ring-shaped multi-layer optimization network is constructed. Multiple cascaded ring-shaped multi-layer optimization networks constitute a multi-scale super-resolution network to obtain super-resolution images of multiple scales corresponding to each super-resolution task sample;
[0016] Step 5: Construct a multi-branch feature extraction network and propose a multi-scale feature fusion mechanism. Two parallel image classifiers form a multi-head classification network to obtain multi-scale classification results and target scale classification results.
[0017] Step 6: Construct a class-guided super-resolution dynamic refinement mechanism, and use the multi-scale classification results to guide the target scale image to learn new knowledge and further reconstruct high-quality images;
[0018] Step 7: Design a multi-task learning strategy to jointly optimize image super-resolution and classification tasks. The multi-scale super-resolution network, multi-head classification network and class-guided super-resolution dynamic refinement mechanism together constitute a multi-task learning model. Image super-resolution and classification tasks are performed at each time step of reverse denoising to achieve joint optimization of the two tasks.
[0019] Step 8: Perform supervised training on the constructed multi-task learning model to obtain the optimal network parameters of the model;
[0020] Step 9: Input all samples into the trained network for image super-resolution and classification operations to obtain the final high spatial resolution hyperspectral image and the corresponding classification result map.
[0021] Further, in step 1, the target area has a low spatial resolution hyperspectral image, and the image is preprocessed:
[0022] Hyperspectral image with low spatial resolution of the target area X L , and perform maximum and minimum normalization on the image. The normalization formula is:
[0023]
[0024] in, represents the pixel value in the original hyperspectral image, and Respectively represent the maximum and minimum values of the hyperspectral image; is the normalized pixel value of the hyperspectral image.
[0025] Further, in step 2, the sample set is generated, and a certain number of samples are selected as the training set I train The remaining samples are used as test set I test ,include:
[0026] (1) The size of the three-dimensional hyperspectral image is H×W×C, where H and W are the height and width of the hyperspectral image, respectively, and C is the number of bands of the hyperspectral image. For the image super-resolution task, the image is cut into non-overlapping data cubes, and a 32×32×C data cube is selected as the super-resolution task input. All samples are constructed into a super-resolution sample set according to the above operation. For the image classification task, the training sample set of the image super-resolution task is In , with the sample point as the center, an image block of 11×11×C pixels is selected as the input of the classification task, and all the sample points are operated as above to form a classification sample set;
[0027] (2) Calculate the training sample set for the image super-resolution task The total number of samples of different categories in the corresponding reference label L i, divide the classification task training sample set according to the proportion of different categories And the test set samples This enables the model to learn the differences between different categories and better adapt to image super-resolution and classification tasks;
[0028] (3) Randomly select 60% of the super-resolution sample set as the super-resolution task training sample set The rest are used as super-resolution task test sample sets According to the proportion of each category, a group of 100 sample points of each category is carefully selected as the classification task training sample set The rest are used as classification task test sample sets Super-resolution task training sample set And classification task training sample set The training set I train , super-resolution task test sample set And classification task test sample set The test set I test .
[0029] Furthermore, step three: design a collaborative iterative super-resolution and classification diffusion model. In the forward process, Gaussian noise is gradually added to the reference image to degenerate the image into a pure noise state, providing a basis for the subsequent reverse denoising process, including:
[0030] (1) Target reference image X H is the original hyperspectral image X L Size 2 n times, and use the following formula to obtain reference images of multiple scales:
[0031] X 0 =[down n-1 (X H ),down n-2 (X H ),…,X H ]
[0032] Among them, X 0 represents the reference image of multiple scales, down(·) represents the 2x downsampling operation, down n (·) means 2 n times downsampling operation;
[0033] (2) In the forward process, the reference image X 0 Gradually add Gaussian noise to degenerate the image into a pure noise state, providing a basis for the subsequent reverse denoising process, which can be expressed by the following formula:
[0034]
[0035] Among them, the time parameter α t ∈(0,1) determines the variance of the Gaussian noise added at each diffusion step, I is the unit matrix, X t and X t-1 represent the state of the diffusion process at time t and time t-1 respectively, X 1:T represents the set of all states from time step t = 1 to t = T, q(X 1:T |X 0 ) represents the image from the reference image X 0 Diffusion to the pure noise image X through T time steps T ,q(X t |X t-1 ) indicates that from X t-1 State spreads to X t state, N(·) represents a normally distributed process;
[0036] Accordingly, given a reference image X 0 , X t The distribution of can be expressed as:
[0037]
[0038] in, γ t represents the cumulative scaling effect of the diffusion process from the initial state to the tth time step;
[0039] Furthermore, for a given X t and X 0 In the case of, through the following algebraic operation, we can get X t-1 The posterior probability distribution of :
[0040]
[0041] Among them, μ and σ 2 Respectively represent the mean and variance of the forward diffusion process, γ t-1 represents the cumulative scaling effect of the diffusion process from the initial state to the t-1th time step.
[0042] Furthermore, in step 4, in the reverse denoising process, a ring-shaped multi-layer optimization network is constructed, and multiple cascaded ring-shaped multi-layer optimization networks constitute a multi-scale super-resolution network to obtain super-resolution images of multiple scales corresponding to each super-resolution task sample, including:
[0043] (1) The constructed ring multi-layer optimization network consists of two parts: an encoder and a decoder. For the first-level ring multi-layer optimization network of the super-resolution task, the input is a sample X of size 32×32×C. Land a noise image of size 64×64×C, each part contains 4 residual blocks, so the feature map sizes obtained by the encoder are 64×64×256, 32×32×512, 16×16×1024, and 8×8×1024 respectively; the feature map sizes obtained by the decoder are 16×16×1024, 32×32×512, 64×64×256 respectively, and the sample X of 64×64×C predicted by the first-level ring multi-layer optimization network L ;
[0044] The residual block consists of two convolutional layers, a regularization layer, and an activation layer; the convolution kernel size of the convolution operation is 3×3, the step size is 2 and 1; the activation function is ReLU;
[0045] (2) Processing noise through a noise level embedding network to improve image quality; encoding the input noise level through a position encoding layer to generate a noise embedding vector; the noise embedding vector is processed through a fully connected layer to generate noise level features for adjusting the residual blocks of each layer in the ring multi-layer optimization network;
[0046] (3) Multiple cascaded annular multi-layer optimization networks constitute a multi-scale super-resolution network to achieve step-by-step super-resolution prediction; the first annular multi-layer optimization network receives the original low-resolution image and the corresponding noise image, and generates a super-resolution image predicted in the first stage; the super-resolution image predicted in the first stage and the corresponding noise image are used as input and passed to the next annular multi-layer optimization network for further super-resolution processing; each subsequent annular multi-layer optimization network receives the output of the previous network and predicts and generates an image with a higher resolution until the final target resolution is reached; through multiple cascaded annular multi-layer optimization networks, the resolution of the image can be gradually improved, and super-resolution images of multiple scales can be predicted and generated. This process can be expressed by the following formula:
[0047]
[0048] Among them, X L represents the original hyperspectral image, Represents a super-resolution image of twice the scale predicted by the super-resolution network, Represents the 2 predicted by the super-resolution network n-1 times the scale of super-resolution images, Represents the 2 predicted by the super-resolution network n times the scale of super-resolution images, represents the super-resolution images of multiple scales predicted by the multi-scale super-resolution network, X con represents the conditional input of each annular multi-layer optimization network, up(·) represents a 2x upsampling operation, cat[·] represents a concatenation operation along the channel dimension, Θ s{·} represents a multi-scale super-resolution network.
[0049] Furthermore, in step 5, a multi-branch feature extraction network is constructed, a multi-scale feature fusion mechanism is proposed, and two parallel image classifiers constitute a multi-head classification network to obtain multi-scale classification results and target scale classification results, including:
[0050] (1) A multi-branch feature extraction network is constructed, and a multi-scale feature fusion mechanism is proposed; the multi-branch feature extraction network is composed of multiple parallel convolution branches, each convolution branch processes super-resolution images of different scales; each convolution branch includes several convolution layers, batch normalization layers and ReLU activation function layers, extracts the spatial-spectral features of each scale super-resolution image, and the obtained feature size is 128×1×1; the extracted multi-scale features are fused through the multi-scale feature fusion mechanism to generate multi-scale fused spatial-spectral features; the multi-branch feature extraction and multi-scale feature fusion can be expressed by the following formula:
[0051]
[0052] Among them, X L represents the original hyperspectral image, Represents the 2 predicted by the super-resolution network n times the super-resolution image, FE(·) represents the feature extraction network, cat(·) represents the feature fusion mechanism, and Represent the output characteristics of each branch, and Respectively represent 2 N-1 Multiple-scale output features, target-scale output features, and multi-scale fusion features;
[0053] (2) and After passing through three fully connected layers, the number of input nodes is set to 128 and (N+1)×128 respectively, and the number of output nodes of each layer is set to 128, 64 and C respectively. Then the softmax function is used to obtain the probability of each category, and the multi-scale classification results and target scale classification results are obtained.
[0054] Furthermore, in step 6, a class-guided super-resolution dynamic refinement mechanism is constructed, and the multi-scale classification results are used to guide the target scale image to learn new knowledge and further reconstruct high-quality images, including:
[0055] (1) At the reverse denoising time step T, the loss contribution calculation is performed on the obtained multi-scale classification results and the target scale classification results, so that the model pays more attention to inconsistent samples; the loss contribution calculation is expressed by the formula:
[0056]
[0057] Among them, M is the total number of categories, Y i,c Indicates whether there are sample points belonging to category c in the multi-scale classification results. represents the probability that the target scale predicts that sample i belongs to class c, γ is the adjustment factor, Indicates the loss contribution consistent with the target scale and multi-scale classification results;
[0058] (2) Using the following formula, Direct the diffusion sampling process to the desired classification label to further refine the super-resolution results:
[0059]
[0060] Among them, s is the classification guidance factor, Represents the last level of the ring multi-layer optimization network Θ in the super-resolution network sN The predicted image, and Respectively represent the input and output of the super-resolution network at time step T, Represented by Θ sN The mean value obtained, σ 2 represents the variance obtained by the forward process, I represents the identity matrix, ω * represents the optimal network parameters of the model, and N(·) represents a normal distribution process.
[0061] Furthermore, step seven: design a multi-task learning strategy for jointly optimizing image super-resolution and classification tasks. The multi-scale super-resolution network, the multi-head classification network and the class-guided super-resolution dynamic refinement mechanism together constitute a multi-task learning model. The image super-resolution and classification tasks are performed at each time step of the reverse denoising to achieve joint optimization of the two tasks, including:
[0062] (1) Design a multi-task learning strategy that jointly optimizes image super-resolution and classification tasks. Integrate the multi-scale super-resolution network, multi-head classification network, and class-oriented super-resolution dynamic refinement mechanism to build a multi-task learning model. At the time step T of reverse denoising, the multi-scale super-resolution network can be expressed as:
[0063]
[0064] Among them, p(X T ) represents the pure noise image X at time step T T It follows a normal distribution with zero mean and unit variance. represents the reverse denoising process, Represented by the multi-scale super-resolution network Θ s The predicted mean, σ 2 represents the variance obtained by the forward process, ω *represents the optimal network parameters of the model;
[0065] The multi-head classification network can be expressed as:
[0066]
[0067] Among them, Y and y n They are respectively represented as multi-scale classification results and target scale classification results, Θ c (·) represents a multi-head classification network, Represents the multi-scale super-resolution network Θ s Predicted super-resolution images at multiple scales;
[0068] Class-oriented super-resolution dynamic refinement mechanism Θ Guide It can be expressed as:
[0069]
[0070] in, and Represent the two states of the target scale image, It is a super-resolution result that needs further refinement;
[0071] Constructed multi-task learning model Θ M It can be expressed as:
[0072]
[0073] (2) In each time step, super-resolution reconstruction and classification tasks of the image are performed simultaneously. The super-resolution task provides more detailed spatial texture information for the classification network, thereby improving the classification performance. At the same time, the classification task uses the class-guided super-resolution dynamic refinement mechanism to provide more detailed optimization directions for the super-resolution network, which is conducive to further refinement of the super-resolution image, thereby achieving joint optimization of the two tasks.
[0074] Furthermore, in step 7, supervised training is performed on the constructed multi-task learning model to obtain the optimal network parameters of the model, including:
[0075] (1) The super-resolution task training sample set with reference images Each time, 4 samples are randomly selected as a batch and input into the multi-scale super-resolution network to be trained, and the image prediction of the training samples is output; the labeled classification task training sample set Each time, 16 samples are randomly selected as a batch and input into the multi-head classification network to be trained, and the label prediction of the training samples is output;
[0076] (2) Use the following loss function formula to calculate the loss function between the predicted image and the reference image, and between the predicted label and the true label of the reference image in the super-resolution and classification tasks respectively:
[0077]
[0078] in, is the final loss function, To rebuild the losses, is the classification loss, λ is the weight coefficient, Represents multiple scale resolution images predicted by the super-resolution network, X 0 yes The corresponding reference image, represents the mean operation, ||·|| 1 represents the L1-norm, M represents the total number of categories, Indicates whether there are sample points in the reference image label that belong to category c. and i,c They represent the target scale and the probability that the multi-scale predicted sample i belongs to class c respectively;
[0079] (3) The network parameters are trained using the adaptive moment estimation algorithm Adam until the network converges to the optimal value and the optimal network parameters are saved.
[0080] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:
[0081] First, based on the diffusion model, the present invention combines the super-resolution network and the classification network in a multi-task learning manner; compared with the existing image super-resolution methods, the present invention reconstructs a high spatial resolution hyperspectral image suitable for the classification task under the guidance of the classification task.
[0082] The present invention jointly optimizes image super-resolution and classification tasks in a continuous iterative process, achieves mutual guidance between the two tasks, gradually reconstructs high-quality images, and improves classification accuracy.
[0083] The present invention proposes a class-oriented super-resolution dynamic refinement mechanism, which ensures that the super-resolution network obtains more detailed optimization directions from the classification task, further reconstructs high-quality images, and improves classification performance.
[0084] Second, the present invention solves the following technical problems:
[0085] (1) Based on the diffusion model, the present invention combines the super-resolution network and the classification network in a multi-task learning manner; compared with the existing image super-resolution methods, the present invention reconstructs a high spatial resolution hyperspectral image suitable for the classification task under the guidance of the classification task.
[0086] (2) The present invention jointly optimizes the image super-resolution and classification tasks in a continuous iterative process, realizes mutual guidance between the two tasks, gradually reconstructs high-quality images, and improves classification accuracy at the same time.
[0087] (3) The present invention proposes a class-guided super-resolution dynamic refinement mechanism, which ensures that the super-resolution network obtains more detailed optimization directions from the classification task, further reconstructs high-quality images, and improves classification performance.
[0088] The present invention has achieved the following significant technical advances:
[0089] Joint optimization of super-resolution and classification tasks: The present invention realizes the joint optimization of super-resolution and classification tasks by designing a collaborative iterative super-resolution and classification diffusion model, so that the two tasks can promote each other and improve the overall performance.
[0090] Multi-scale super-resolution reconstruction: By constructing a ring-shaped multi-layer optimization network and adopting a multi-scale super-resolution strategy, the spatial resolution of hyperspectral images is significantly improved, more image details are retained, and the image reconstruction quality is improved.
[0091] Efficient feature fusion and classification: A multi-branch feature extraction network and a multi-scale feature fusion mechanism are proposed, and combined with a multi-head classification network, efficient classification of hyperspectral images is achieved, improving classification accuracy.
[0092] Class-guided super-resolution dynamic refinement: Through the class-guided super-resolution dynamic refinement mechanism, the multi-scale classification results are used to guide the reconstruction of the target scale image, further improving the quality and detail expression of the reconstructed image.
[0093] Multi-task learning strategy: A multi-task learning strategy was designed to perform image super-resolution and classification tasks at each time step of reverse denoising, achieving simultaneous optimization of the two tasks and improving the overall performance and robustness of the model.
[0094] Improve processing efficiency: By co-processing super-resolution and classification tasks, the computing resources and time required to process each task separately are reduced, and the processing efficiency of the system is improved.
[0095] Third, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:
[0096] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:
[0097] The hyperspectral image super-resolution and classification joint optimization method based on the diffusion model proposed in the present invention can significantly improve the spatial resolution and classification accuracy of hyperspectral images, which has important application value in the fields of remote sensing image processing, environmental monitoring, agricultural resource management, and military reconnaissance. By improving the spatial resolution and classification accuracy of the image, the present invention helps to obtain more accurate and detailed ground object information, thereby improving the scientificity and effectiveness of related decisions. In addition, the diffusion model adopted by the present invention has good generalization ability and stability, is easy to promote and apply, and can significantly reduce the time cost of data processing and analysis, bringing significant economic benefits and social value.
[0098] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:
[0099] The present invention solves the problem of joint optimization of hyperspectral image super-resolution and classification, especially in the current hyperspectral image processing, only using the image super-resolution task as the preprocessing stage of the classification task often leads to information loss and insufficient classification accuracy. The present invention combines the super-resolution network and the classification network in a multi-task learning manner based on the diffusion model, realizes the joint optimization of hyperspectral image super-resolution and classification, overcomes the limitations of traditional methods, fills the gap in the joint optimization scheme of hyperspectral image super-resolution and classification at home and abroad, and provides an innovative solution for hyperspectral image processing.
[0100] (3) The technical solution of the present invention solves the technical problems that people have been eager to solve but have never been able to solve successfully:
[0101] The present invention, by introducing a diffusion model, realizes for the first time the joint optimization of hyperspectral image super-resolution and classification, and effectively solves two core problems in hyperspectral image processing for a long time: high spatial resolution image reconstruction and high-precision classification. Traditionally, hyperspectral image super-resolution is a preprocessing technology for classification, and the high-resolution hyperspectral images obtained by super-resolution are crucial to improving the classification accuracy of classification tasks. However, existing image super-resolution methods usually ignore the requirements of classification and cannot effectively generate high-quality images that meet classification tasks, thus affecting the interpretation performance. By utilizing the characteristics of the diffusion model, the present invention not only improves the spatial resolution of the image, but also improves the accuracy of classification, meeting people's demand for high-precision hyperspectral image processing. This method is superior to the existing technology in both processing efficiency and effect, and successfully solves the technical problems that have long been unresolved in the industry.
[0102] (4) The technical solution of the present invention overcomes technical prejudice:.
[0103] The hyperspectral image super-resolution and classification joint optimization method based on the diffusion model proposed in the present invention overcomes the technical bias in traditional hyperspectral image processing methods. Traditionally, hyperspectral image super-resolution is a preprocessing technology for classification, and the high spatial resolution hyperspectral images obtained by super-resolution are crucial to improving the classification accuracy of classification tasks. However, existing image super-resolution methods usually ignore the requirements of classification and cannot effectively generate high-quality images that meet the classification tasks, thus affecting the interpretation performance. The present invention breaks this technical bias by introducing a diffusion model and jointly optimizing the super-resolution and classification tasks, making full use of the spatial and spectral information in the hyperspectral images, thereby greatly improving the overall performance of image processing. In addition, the application of the diffusion model overcomes the limitations of traditional algorithms that rely on manually designed features and parameter tuning, and fully demonstrates the advantages of modern deep learning technology in hyperspectral image processing. This innovative joint optimization strategy not only improves processing efficiency and accuracy, but also provides new ideas and methods for the field of hyperspectral image processing.
[0104] Fourth, the main parameters, algorithms and mathematical models of the present invention solve technical problems and make significant technological progress.
[0105] 1. Preprocessing and normalization of hyperspectral images
[0106] Main parameters: In step 1, for the hyperspectral image with low spatial resolution in the target area, the maximum and minimum normalization formula is used to preprocess the image.
[0107] Technical problems solved: In the existing technology, the spatial resolution of hyperspectral images is low, which directly affects the subsequent processing effect of the image. Normalization processing can standardize image data so that it is within the same scale range, improving the applicability and stability of image processing algorithms.
[0108] Technological progress: The normalization formula effectively standardizes the image data, enhances the robustness of the image processing algorithm, improves the consistency of image processing, and lays the foundation for efficient processing in subsequent steps.
[0109] 2. Sample set generation and classification task training
[0110] Main parameters: In step 2, the size of the three-dimensional hyperspectral image is H×W×C. The training set and test set for the super-resolution task and classification task are constructed by cutting the image and selecting samples.
[0111] Technical problems solved: In existing technologies, the generation and division of sample sets lacks systematicity and scientificity, resulting in poor training model results. By reasonably dividing the sample sets, the model can learn the differences between different categories, thereby better adapting to image super-resolution and classification tasks.
[0112] Technological progress: Careful selection and division of sample sets ensures a balanced distribution of training and test data sets, which improves the effect and generalization ability of model training, thereby improving the accuracy and stability of hyperspectral image super-resolution and classification.
[0113] 3. Collaborative Iterative Super-resolution and Classification Diffusion Model
[0114] Algorithm and mathematical model: In step three, a collaborative iterative super-resolution and classification diffusion model is designed. By adding Gaussian noise in the forward direction and then performing reverse denoising, efficient image super-resolution prediction and classification tasks can be achieved.
[0115] Technical problem solved: Traditional super-resolution and classification methods are processed separately, resulting in low information utilization and unsatisfactory results. The collaborative iteration model can perform super-resolution and classification simultaneously under a unified framework, improving information utilization.
[0116] Technological progress: The collaborative iterative model gradually adds Gaussian noise in the forward process, causing the image to degenerate into a pure noise state, providing a basis for reverse denoising. Through joint optimization, it achieves mutual promotion of super-resolution and classification, significantly improving image quality and classification accuracy.
[0117] 4. Multi-task learning and dynamic refinement mechanism
[0118] Algorithm and mathematical model: In steps six and seven, a multi-task learning strategy is designed to jointly optimize image super-resolution and classification tasks, and a class-guided super-resolution dynamic refinement mechanism is constructed. Through the joint optimization of the multi-scale super-resolution network and the multi-head classification network, further refinement and high-quality reconstruction of the image are achieved.
[0119] Technical Problem Solved: In the existing technology, super-resolution reconstruction and classification tasks are performed separately, which fails to fully utilize the relationship between the tasks. The multi-task learning strategy integrates the two tasks and improves the overall performance.
[0120] Technological progress: Through the class-guided super-resolution dynamic refinement mechanism, the classification results are used to guide the further refinement of the super-resolution task, thereby improving the resolution and quality of the image; the multi-task learning strategy optimizes the synergy of the two tasks, making the super-resolution and classification results more accurate, further improving the overall performance of hyperspectral image processing.
[0121] The combination of these technical solutions not only solves the problems of low image quality and poor classification accuracy in existing technologies, but also significantly improves the overall effect and application value of hyperspectral image processing through scientific algorithms and mathematical models. BRIEF DESCRIPTION OF THE DRAWINGS
[0122] Figure 1 It is a flow chart of a joint optimization method for hyperspectral image super-resolution and classification based on a diffusion model provided by an embodiment of the present invention.
[0123] Figure 2 It is a schematic diagram of a joint optimization method for hyperspectral image super-resolution and classification based on a diffusion model provided in an embodiment of the present invention.
[0124] Figure 3 These are the result diagrams of different super-resolution methods provided by embodiments of the present invention; (a) is a reference image; (b) is a result diagram of the Bicubic method; (c) is a result diagram of the SRCNN method; (d) is a result diagram of the EDSR method; (e) is a result diagram of the SSPSR method; (f) is a result diagram of the MCNet method; (g) is a result diagram of the ERCSR method; (h) is a result diagram of the MSDformer method; and (i) is a result diagram of the present invention.
[0125] Figure 4 These are result diagrams of different classification methods provided by embodiments of the present invention; (a) is a pseudo-true color image; (b) is a Ground-truth standard image; (c) is a result diagram of the SSRN method; (d) is a result diagram of the DBDA method; (e) is a result diagram of the A2S2K-ResNet method; (f) is a result diagram of the pResNet method; (g) is a result diagram of the SSFTT method; (h) is a result diagram of the GMANet method; and (i) is a result diagram of the present invention. DETAILED DESCRIPTION
[0126] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0127] In view of the problems existing in the prior art, the present invention provides a hyperspectral image super-resolution and classification joint optimization method based on a diffusion model. The present invention is described in detail below with reference to the accompanying drawings.
[0128] like Figure 1 As shown, the hyperspectral image super-resolution and classification joint optimization method based on the diffusion model provided by the embodiment of the present invention includes the following steps:
[0129] S101, inputting a hyperspectral image with low spatial resolution of the target area and preprocessing the image;
[0130] S102, generating a sample set, selecting a certain number of samples as a training set, and the remaining samples as a test set;
[0131] S103, designing a collaborative iterative super-resolution and classification diffusion model, in the forward process, gradually adding Gaussian noise to the reference image, so that the image is degraded to a pure noise state, providing a basis for the subsequent reverse denoising process;
[0132] S104, in the reverse denoising process, constructing a ring-shaped multi-layer optimization network, wherein multiple cascaded ring-shaped multi-layer optimization networks constitute a multi-scale super-resolution network, and obtaining super-resolution images of multiple scales corresponding to each super-resolution task sample;
[0133] S105, construct a multi-branch feature extraction network, propose a multi-scale feature fusion mechanism, and two parallel image classifiers form a multi-head classification network to obtain multi-scale classification results and target scale classification results;
[0134] S106, constructing a class-guided super-resolution dynamic refinement mechanism, guiding the target scale image to learn new knowledge based on the multi-scale classification results, and further reconstructing a high-quality image;
[0135] S107, design a multi-task learning strategy for jointly optimizing image super-resolution and classification tasks. The multi-scale super-resolution network, the multi-head classification network and the class-guided super-resolution dynamic refinement mechanism together constitute a multi-task learning model, which performs image super-resolution and classification tasks at each time step of reverse denoising to achieve joint optimization of the two tasks;
[0136] S108, performing supervised training on the constructed multi-task learning model to obtain the optimal network parameters of the model;
[0137] S109, input all samples into the trained network for image super-resolution and classification operations to obtain the final high spatial resolution hyperspectral image and the corresponding classification result map.
[0138] The principle diagram of the hyperspectral image super-resolution and classification joint optimization method based on diffusion model provided by the present invention is as follows: Figure 2 shown.
[0139] The technical solution of the present invention is further described below in conjunction with embodiments.
[0140] like Figure 1 As shown, the hyperspectral image super-resolution and classification joint optimization method based on the diffusion model provided by the embodiment of the present invention has the following implementation process:
[0141] (1) Input a hyperspectral image X with low spatial resolution of the target area L , and perform maximum and minimum normalization on the image. The normalization formula is:
[0142]
[0143] in, represents the pixel value in the original hyperspectral image, and Respectively represent the maximum and minimum values of the hyperspectral image; is the normalized pixel value of the hyperspectral image.
[0144] (2) Generate a sample set and select training sample set I train And test sample set I test ;
[0145] (2a) The size of the three-dimensional hyperspectral image is H×W×C, where H and W are the height and width of the hyperspectral image, respectively, and C is the number of bands of the hyperspectral image. For the image super-resolution task, the image is cut into non-overlapping data cubes, and a 32×32×C data cube is selected as the super-resolution task input. All samples are constructed into a super-resolution sample set according to the above operation. For the image classification task, the training sample set of the image super-resolution task is In , with the sample point as the center, an image block of 11×11×C pixels is selected as the input of the classification task, and all the sample points are operated as above to form a classification sample set;
[0146] (2b) Calculate the training sample set for the image super-resolution task The total number of samples of different categories in the corresponding reference label L i , divide the classification task training sample set according to the proportion of different categories And the test set samples This enables the model to learn the differences between different categories and better adapt to image super-resolution and classification tasks;
[0147] (2c) Randomly select 60% of the super-resolution sample set as the super-resolution task training sample set The rest are used as super-resolution task test sample sets According to the proportion of each category, a group of 100 sample points of each category is carefully selected as the classification task training sample set The rest are used as classification task test sample sets Super-resolution task training sample set And classification task training sample set The training set I train , super-resolution task test sample set And classification task test sample set The test set I test .
[0148] (3) Design a collaborative iterative super-resolution and classification diffusion model. In the forward process, Gaussian noise is gradually added to the reference image to degenerate the image into a pure noise state, providing a basis for the subsequent reverse denoising process.
[0149] (3a) Target reference image X H is the original hyperspectral image X L Size 2 n times, and use the following formula to obtain reference images of multiple scales:
[0150] X 0 =[down n-1 (X H ),down n-2 (X H ),…,X H ]
[0151] Among them, X 0 represents the reference image of multiple scales, down(·) represents the 2x downsampling operation, down n (·) means 2 n times downsampling operation;
[0152] (3b) In the forward process, for the reference image X 0 Gradually add Gaussian noise to degenerate the image into a pure noise state, providing a basis for the subsequent reverse denoising process, which can be expressed by the following formula:
[0153]
[0154] Among them, the time parameter α t ∈(0,1) determines the variance of the Gaussian noise added at each diffusion step, I is the unit matrix, X t and X t-1 represent the state of the diffusion process at time t and time t-1 respectively, X 1:T represents the set of all states from time step t = 1 to t = T, q(X 1:T |X 0 ) represents the image from the reference image X 0 Diffusion to the pure noise image X through T time steps T ,q(X t |X t-1 ) indicates that from X t-1 State spreads to X t state, represents a normally distributed process;
[0155] Accordingly, given a reference image X 0 , X t The distribution of can be expressed as:
[0156]
[0157] in, γ t represents the cumulative scaling effect of the diffusion process from the initial state to the tth time step;
[0158] Furthermore, for a given X t and X 0 In the case of, through the following algebraic operation, we can get Xt-1 The posterior probability distribution of :
[0159]
[0160] Among them, μ and σ 2 Respectively represent the mean and variance of the forward diffusion process, γ t-1 represents the cumulative scaling effect of the diffusion process from the initial state to the t-1th time step.
[0161] (4) In the reverse denoising process, a ring-shaped multi-layer optimization network is constructed. Multiple cascaded ring-shaped multi-layer optimization networks constitute a multi-scale super-resolution network to obtain super-resolution images of multiple scales corresponding to each super-resolution task sample, including:
[0162] (4a) The constructed ring multi-layer optimization network consists of two parts: an encoder and a decoder. For the super-resolution task, the first-level ring multi-layer optimization network input is a sample X of size 32×32×C. L and a noise image of size 64×64×C, each part contains 4 residual blocks, so the feature map sizes obtained by the encoder are 64×64×256, 32×32×512, 16×16×1024, and 8×8×1024 respectively; the feature map sizes obtained by the decoder are 16×16×1024, 32×32×512, 64×64×256 respectively, and the sample X of 64×64×C predicted by the first-level ring multi-layer optimization network L ;
[0163] The residual block consists of two convolutional layers, a regularization layer, and an activation layer; the convolution kernel size of the convolution operation is 3×3, the step size is 2 and 1; the activation function is ReLU;
[0164] (4b) Processing noise through a noise level embedding network to improve image quality; encoding the input noise level through a position encoding layer to generate a noise embedding vector; the noise embedding vector is processed through a fully connected layer to generate a noise level feature for adjusting the residual blocks of each layer in the ring multi-layer optimization network;
[0165] (4c) Multiple cascaded annular multi-layer optimization networks constitute a multi-scale super-resolution network to achieve step-by-step super-resolution prediction; the first annular multi-layer optimization network receives the original low-resolution image and the corresponding noise image, and generates a super-resolution image predicted in the first stage; the super-resolution image predicted in the first stage and the corresponding noise image are used as input and passed to the next annular multi-layer optimization network for further super-resolution processing; each subsequent annular multi-layer optimization network receives the output of the previous network and predicts and generates an image with a higher resolution until the final target resolution is reached; through multiple cascaded annular multi-layer optimization networks, the resolution of the image can be gradually improved, and super-resolution images of multiple scales can be predicted and generated. This process can be expressed by the following formula:
[0166]
[0167] Among them, X L represents the original hyperspectral image, Represents a super-resolution image of twice the scale predicted by the super-resolution network, Represents the 2 predicted by the super-resolution network n-1 times the scale of super-resolution images, Represents the 2 predicted by the super-resolution network n times the scale of super-resolution images, represents the super-resolution images of multiple scales predicted by the multi-scale super-resolution network, X con represents the conditional input of each annular multi-layer optimization network, up(·) represents a 2x upsampling operation, cat[·] represents a concatenation operation along the channel dimension, Θ s {·} represents a multi-scale super-resolution network.
[0168] (5) Construct a multi-branch feature extraction network and propose a multi-scale feature fusion mechanism. Two parallel image classifiers form a multi-head classification network to obtain multi-scale classification results and target scale classification results, including:
[0169] (5a) A multi-branch feature extraction network is constructed, and a multi-scale feature fusion mechanism is proposed; the multi-branch feature extraction network is composed of multiple parallel convolution branches, each convolution branch processes super-resolution images of different scales; each convolution branch includes several convolution layers, batch normalization layers and ReLU activation function layers, extracts the spatial-spectral features of each scale super-resolution image, and the obtained feature size is 128×1×1; the extracted multi-scale features are fused through the multi-scale feature fusion mechanism to generate multi-scale fused spatial-spectral features; the multi-branch feature extraction and multi-scale feature fusion can be expressed by the following formula:
[0170]
[0171] Among them, X Lrepresents the original hyperspectral image, Represents the 2 predicted by the super-resolution network n times the super-resolution image, FE(·) represents the feature extraction network, cat(·) represents the feature fusion mechanism, and Represent the output characteristics of each branch, and Respectively represent 2 N-1 Multiple-scale output features, target-scale output features, and multi-scale fusion features;
[0172] (5b) and After passing through three fully connected layers, the number of input nodes is set to 128 and (N+1)×128 respectively, and the number of output nodes of each layer is set to 128, 64 and C respectively. Then the softmax function is used to obtain the probability of each category, and the multi-scale classification results and target scale classification results are obtained.
[0173] (6) Construct a class-guided super-resolution dynamic refinement mechanism, which uses the multi-scale classification results to guide the target scale image to learn new knowledge and further reconstruct high-quality images, including:
[0174] (6a) At the reverse denoising time step T, the loss contribution calculation is performed on the obtained multi-scale classification results and the target scale classification results, so that the model pays more attention to inconsistent samples; the loss contribution calculation is expressed by the formula:
[0175]
[0176] Among them, M is the total number of categories, Y i,c Indicates whether there are sample points belonging to category c in the multi-scale classification results. represents the probability that the target scale predicts that sample i belongs to class c, γ is the adjustment factor, Indicates the loss contribution consistent with the target scale and multi-scale classification results;
[0177] (6b) Using the following formula, Direct the diffusion sampling process to the desired classification label to further refine the super-resolution results:
[0178]
[0179] Among them, s is the classification guidance factor, Represents the last level of the ring multi-layer optimization network Θ in the super-resolution network sN The predicted image, and Respectively represent the input and output of the super-resolution network at time step T, Represented by Θ sNThe mean value obtained, σ 2 represents the variance obtained by the forward process, I represents the identity matrix, ω * represents the optimal network parameters of the model, represents a normally distributed process.
[0180] (7) Design a multi-task learning strategy for jointly optimizing image super-resolution and classification tasks. The multi-scale super-resolution network, multi-head classification network, and class-guided super-resolution dynamic refinement mechanism together constitute a multi-task learning model. Image super-resolution and classification tasks are performed at each time step of reverse denoising to achieve joint optimization of the two tasks, including:
[0181] (7a) Design a multi-task learning strategy that jointly optimizes image super-resolution and classification tasks. Integrate the multi-scale super-resolution network, multi-head classification network, and class-oriented super-resolution dynamic refinement mechanism to build a multi-task learning model. At the time step T of reverse denoising, the multi-scale super-resolution network can be expressed as:
[0182]
[0183] Among them, p(X T ) represents the pure noise image X at time step T T It follows a normal distribution with zero mean and unit variance. represents the reverse denoising process, Represented by the multi-scale super-resolution network Θ s The predicted mean, σ 2 represents the variance obtained by the forward process, ω * represents the optimal network parameters of the model;
[0184] The multi-head classification network can be expressed as:
[0185]
[0186] Among them, Y and y n They are respectively represented as multi-scale classification results and target scale classification results, Θ c (·) represents a multi-head classification network, Represents the multi-scale super-resolution network Θ s Predicted super-resolution images at multiple scales;
[0187] Class-oriented super-resolution dynamic refinement mechanism Θ Guide It can be expressed as:
[0188]
[0189] in, and Represent the two states of the target scale image, It is a super-resolution result that needs further refinement;
[0190] Constructed multi-task learning model Θ M It can be expressed as:
[0191]
[0192] (7b) In each time step, the image super-resolution reconstruction and classification tasks are performed simultaneously. The super-resolution task provides more detailed spatial texture information for the classification network, thereby improving the classification performance. At the same time, the classification task uses the class-oriented super-resolution dynamic refinement mechanism to provide the super-resolution network with more detailed optimization directions, which is conducive to further refinement of the super-resolution image, thereby achieving joint optimization of the two tasks.
[0193] (8) Perform supervised training on the constructed multi-task learning model to obtain the optimal network parameters of the model, including:
[0194] (8a) The super-resolution task training sample set with reference images Each time, 4 samples are randomly selected as a batch and input into the multi-scale super-resolution network to be trained, and the image prediction of the training samples is output; the labeled classification task training sample set Each time, 16 samples are randomly selected as a batch and input into the multi-head classification network to be trained, and the label prediction of the training samples is output;
[0195] (8b) The following loss function formula is used to calculate the loss function between the predicted image and the reference image and between the predicted label and the true label of the reference image in the super-resolution and classification tasks respectively:
[0196]
[0197] in, is the final loss function, To rebuild the losses, is the classification loss, λ is the weight coefficient, Represents multiple scale resolution images predicted by the super-resolution network, X 0 yes The corresponding reference image, represents the mean operation, ||·|| 1 represents the L1-norm, M represents the total number of categories, Indicates whether there are sample points in the reference image label that belong to category c. and i,c They represent the target scale and the probability that the multi-scale predicted sample i belongs to class c respectively;
[0198] (8c) Use the adaptive moment estimation algorithm (Adam) to train the network parameters and traverse the entire super-resolution task training sample set It is called a round of super-resolution task training, which traverses the entire classification task training sample set during the training process. This is called a round of classification task training. First, 10,000 rounds of super-resolution task training are performed, followed by 300 rounds of classification task training, with the learning rate set to 0.0001. Finally, the parameters of the class-oriented super-resolution dynamic refinement mechanism are adjusted. The network converges to the optimal value and saves the optimal network parameters.
[0199] (9) All samples are input into the trained network for image super-resolution and classification operations to obtain the final high spatial resolution hyperspectral image and the corresponding classification result map.
[0200] 2. Application Examples: In order to prove the creativity and technical value of the technical solution of the present invention, this section provides application examples of the technical solution of the claims on specific products or related technologies.
[0201] The following application examples demonstrate the application of the present invention in multiple real-world tasks, including agricultural production, urban planning, marine protection, forest protection, geological resource protection, medical image analysis, unmanned driving and intelligent transportation, video surveillance and security, document and fingerprint recognition, cultural relics protection and restoration, and other fields.
[0202] The technical effects of the present invention are described in detail below in conjunction with simulation experiments.
[0203] 1. Simulation experiment conditions:
[0204] The hardware platform of the simulation experiment of the present invention is: the processor is Inter i9-10900X CPU, and the graphics card is Nvidia GeForce RTX 3090.
[0205] The software platforms for the simulation experiment of the present invention are: Linux18.06 operating system, python3.7 and pytorch1.12.
[0206] The hyperspectral data set used in the simulation experiment of the present invention was taken by the Reflective Optics Spectrographic Imaging System (ROSIS-03) in Pavia, northern Italy in 2003. The image size is 1096×715×102, the spatial resolution is 1.3m, and it covers 102 bands from 430nm to 860nm. As shown in Table 1, the data set contains 9 categories and also shows the division of the classification task training sample points.
[0207] Table 1 Detailed categories and sample numbers of the Pavia Center dataset
[0208]
[0209] 2. Experimental content and results analysis:
[0210] In order to verify that the present invention has good super-resolution performance, six evaluation indicators are used, including peak signal-to-noise ratio (PSNR), cross correlation (CC), spectral angle matching (SAM), root mean square error (RMSE), error relative global accuracy (ERGAS) and structural similarity (SSIM). The higher the PSNR, CC and SSIM, the better, the ideal value of PSNR is infinity, the ideal values of CC and SSIM are both 1, the smaller the SAM, ERGAS and RMSE, the better, and the ideal values of SAM, ERGAS and RMSE are all 0.
[0211] In order to verify that the present invention has good classification accuracy, three evaluation indicators are used: overall accuracy (OA), average accuracy (AA), and Kappa coefficient (Kappa). OA represents the total number of correctly classified pixels divided by the total number of pixels. The closer this value is to 1, the better. AA represents the average of the sum of the recall rates of each category. Kappa represents the ratio of the error reduction produced by classification to completely random classification. The closer this value is to 1, the better.
[0212] Experiment 1: The image is super-resolved using the present invention and the existing comparison algorithm. The performance indicators of the obtained results are shown in Table 2. Figure 3 Visualization results of all methods are shown.
[0213] Table 2 Performance indicators of image super-resolution by the present invention and existing comparison algorithms
[0214]
[0215] It can be seen from Table 2 that compared with other algorithms, the super-resolution results of the present invention perform best in all indicators. Compared with MSDformer, which has the best effect among the comparison algorithms, the present invention uses a diffusion model to gradually refine the image, which has more advantages in PSNR, proving that the present invention can perform better image super-resolution.
[0216] Experiment 2: The image is classified using the present invention and the existing comparison algorithm. The performance indicators of the obtained results are shown in Table 3. Figure 4 Visualization results of all methods are shown.
[0217] Table 3 Performance indicators of the present invention and existing comparison algorithms for image classification
[0218]
[0219] It can be seen from Table 3 that compared with other algorithms, the OA, AA and Kappa of the present invention reached 99.12%, 98.88% and 98.75% respectively, which were improved by 0.07%, 0.8% and 0.09% respectively compared with GMNet which had the best effect among the comparison algorithms, proving that the present invention can better perform land cover classification.
[0220] In summary, the present invention combines the super-resolution network and the classification network in a multi-task learning manner through a diffusion model, and introduces a class-guided super-resolution dynamic refinement mechanism. In the continuous iterative process, the image super-resolution and classification tasks are jointly optimized, and the mutual guidance between the two tasks is achieved, thereby gradually reconstructing high-quality images and improving the classification accuracy.
[0221] The present invention can be applied in many real-world tasks, including agricultural production, urban planning, geological resource protection, medical image analysis, unmanned driving and intelligent transportation, video surveillance and security, document and fingerprint recognition, cultural relics protection and restoration, etc. In these applications, there are high requirements for image quality and classification accuracy. Through the method of the present invention, the mutual guidance of image super-resolution and classification tasks can be achieved, high-quality images can be gradually reconstructed, and classification accuracy can be improved, thereby providing a better solution.
[0222] The technical effects of the present invention are described in detail below in conjunction with experiments.
[0223] To further demonstrate the effectiveness of the present invention, the performance of the present invention and its two variants "Va1" and "Va2" are compared. Va1 removes the class-oriented super-resolution dynamic refinement mechanism and replaces multiple cascaded ring-shaped multi-layer optimization networks in the super-resolution network with a single ring-shaped multi-layer optimization network; Va2 removes the class-oriented super-resolution dynamic refinement mechanism. Both "Va1" and "Va2" first use the super-resolution network for super-resolution operations, and then use the classification network to classify the super-resolution images. The experimental results are shown in Tables 4 and 5.
[0224] Table 4 Comparison of performance indicators of the present invention and its corresponding ablation experiment in image super-resolution tasks
[0225]
[0226] Table 5 Comparison of performance indicators of the present invention and its corresponding ablation experiment in image classification tasks
[0227]
[0228] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When the use is implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (e.g., infrared, wireless, microwave, etc.) mode) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.
[0229] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.
Claims
1. A hyperspectral image super-resolution and classification joint optimization method based on diffusion model, characterized in that: The following steps are involved: Step 1: Input a hyperspectral image with low spatial resolution of the target area and preprocess the image; Step 2: Generate a sample set, select a certain number of samples as the training set, and the remaining samples as the test set; Step 3: Design a collaborative iterative super-resolution and classification diffusion model. In the forward process, gradually add Gaussian noise to the reference image to degenerate the image into a pure noise state, providing a basis for the subsequent reverse denoising process. Step 4: In the reverse denoising process, a ring-shaped multi-layer optimization network is constructed. Multiple cascaded ring-shaped multi-layer optimization networks constitute a multi-scale super-resolution network to obtain super-resolution images of multiple scales corresponding to each super-resolution task sample; Step 5: Construct a multi-branch feature extraction network and propose a multi-scale feature fusion mechanism. Two parallel image classifiers form a multi-head classification network to obtain multi-scale classification results and target scale classification results. Step 6: Construct a class-guided super-resolution dynamic refinement mechanism, and use the multi-scale classification results to guide the target scale image to learn new knowledge and further reconstruct high-quality images; Step 7: Design a multi-task learning strategy to jointly optimize image super-resolution and classification tasks. The multi-scale super-resolution network, multi-head classification network and class-guided super-resolution dynamic refinement mechanism together constitute a multi-task learning model. Image super-resolution and classification tasks are performed at each time step of reverse denoising to achieve joint optimization of the two tasks. Step 8: Perform supervised training on the constructed multi-task learning model to obtain the optimal network parameters of the model; Step 9: Input all samples into the trained network for image super-resolution and classification operations to obtain the final high spatial resolution hyperspectral image and the corresponding classification result map.
2. The hyperspectral image super-resolution and classification joint optimization method based on diffusion model as claimed in claim 1, characterized in that: In step 1, the target area has a low spatial resolution hyperspectral image, and the image is preprocessed: Hyperspectral image with low spatial resolution of the target area X L , and perform maximum and minimum normalization on the image. The normalization formula is: in, represents the pixel value in the original hyperspectral image, and Respectively represent the maximum and minimum values of the hyperspectral image; is the normalized pixel value of the hyperspectral image.
3. The hyperspectral image super-resolution and classification joint optimization method based on diffusion model as claimed in claim 1, characterized in that: In step 2, the sample set is generated, and a certain number of samples are selected as the training set I train The remaining samples are used as test set I test ,include: (1) The size of the three-dimensional hyperspectral image is H×W×C, where H and W are the height and width of the hyperspectral image, respectively, and C is the number of bands of the hyperspectral image. For the image super-resolution task, the image is cut into non-overlapping data cubes, and a 32×32×C data cube is selected as the super-resolution task input. All samples are constructed into a super-resolution sample set according to the above operation. For the image classification task, the training sample set of the image super-resolution task is In , with the sample point as the center, an image block of 11×11×C pixels is selected as the input of the classification task, and all the sample points are operated as above to form a classification sample set; (2) Calculate the training sample set for the image super-resolution task The total number of samples of different categories in the corresponding reference label L i , divide the classification task training sample set according to the proportion of different categories And the test set samples This enables the model to learn the differences between different categories and better adapt to image super-resolution and classification tasks; (3) Randomly select 60% of the super-resolution sample set as the super-resolution task training sample set The rest are used as super-resolution task test sample sets According to the proportion of each category, a group of 100 sample points of each category is carefully selected as the classification task training sample set The rest are used as classification task test sample sets Super-resolution task training sample set And classification task training sample set The training set I train , super-resolution task test sample set And classification task test sample set The test set I test .
4. The hyperspectral image super-resolution and classification joint optimization method based on diffusion model as claimed in claim 1, characterized in that: Step 3: Design a collaborative iterative super-resolution and classification diffusion model. In the forward process, gradually add Gaussian noise to the reference image to degenerate the image into a pure noise state, providing a basis for the subsequent reverse denoising process, including: (1) Target reference image X H is the original hyperspectral image X L Size 2 n times, and use the following formula to obtain reference images of multiple scales: X0=[down n-1 (X H ),down n-2 (X H ),...,X H ] Among them, X0 represents the reference image of multiple scales, down(·) represents the 2x downsampling operation, and down n (·) means 2 n times downsampling operation; (2) In the forward process, Gaussian noise is gradually added to the reference image X0 to degenerate the image into a pure noise state, which provides a basis for the subsequent reverse denoising process and can be expressed by the following formula: Among them, the time parameter α t ∈(0,1) determines the variance of the Gaussian noise added at each diffusion step, I is the unit matrix, X t and X t-1 represent the state of the diffusion process at time t and time t-1 respectively, X 1:T represents the set of all states from time step t = 1 to t = T, q(X 1:T |X0) represents the diffusion from the reference image X0 to the pure noise image X through T time steps T ,q(X t |X t-1 ) indicates that from X t-1 State spreads to X t state, represents a normally distributed process; Accordingly, given a reference image X0, X t The distribution of can be expressed as: in, γ t represents the cumulative scaling effect of the diffusion process from the initial state to the tth time step; Furthermore, for a given X t and X0, we can get X by the following algebraic operation: t-1 The posterior probability distribution of : Among them, μ and σ 2 Respectively represent the mean and variance of the forward diffusion process, γ t-1 represents the cumulative scaling effect of the diffusion process from the initial state to the t-1th time step.
5. The hyperspectral image super-resolution and classification joint optimization method based on diffusion model as claimed in claim 1, characterized in that: In step 4, in the reverse denoising process, a ring-shaped multi-layer optimization network is constructed. Multiple cascaded ring-shaped multi-layer optimization networks constitute a multi-scale super-resolution network to obtain super-resolution images of multiple scales corresponding to each super-resolution task sample, including: (1) The constructed ring multi-layer optimization network consists of two parts: an encoder and a decoder. For the first-level ring multi-layer optimization network of the super-resolution task, the input is a sample X of size 32×32×C. L and a noise image of size 64×64×C, each part contains 4 residual blocks, so the feature map sizes obtained by the encoder are 64×64×256, 32×32×512, 16×16×1024, and 8×8×1024 respectively; the feature map sizes obtained by the decoder are 16×16×1024, 32×32×512, 64×64×256 respectively, and the sample X of 64×64×C predicted by the first-level ring multi-layer optimization network L ; The residual block consists of two convolutional layers, a regularization layer, and an activation layer; the convolution kernel size of the convolution operation is 3×3, the step size is 2 and 1; the activation function is ReLU; (2) Processing noise through a noise level embedding network to improve image quality; encoding the input noise level through a position encoding layer to generate a noise embedding vector; the noise embedding vector is processed through a fully connected layer to generate noise level features for adjusting the residual blocks of each layer in the ring multi-layer optimization network; (3) Multiple cascaded annular multi-layer optimization networks constitute a multi-scale super-resolution network to achieve step-by-step super-resolution prediction; the first annular multi-layer optimization network receives the original low-resolution image and the corresponding noise image, and generates a super-resolution image predicted in the first stage; the super-resolution image predicted in the first stage and the corresponding noise image are used as input and passed to the next annular multi-layer optimization network for further super-resolution processing; each subsequent annular multi-layer optimization network receives the output of the previous network and predicts and generates an image with a higher resolution until the final target resolution is reached; through multiple cascaded annular multi-layer optimization networks, the resolution of the image can be gradually improved, and super-resolution images of multiple scales can be predicted and generated. This process can be expressed by the following formula: Among them, X L represents the original hyperspectral image, Represents a super-resolution image of twice the scale predicted by the super-resolution network, Represents the 2 predicted by the super-resolution network n-1 times the scale of super-resolution images, Represents the 2 predicted by the super-resolution network n times the scale of super-resolution images, represents the super-resolution images of multiple scales predicted by the multi-scale super-resolution network, X con represents the conditional input of each annular multi-layer optimization network, up(·) represents a 2x upsampling operation, cat[·] represents a concatenation operation along the channel dimension, Θ s {·} represents a multi-scale super-resolution network.
6. The hyperspectral image super-resolution and classification joint optimization method based on diffusion model as claimed in claim 1, characterized in that: In step 5, a multi-branch feature extraction network is constructed, a multi-scale feature fusion mechanism is proposed, and two parallel image classifiers constitute a multi-head classification network to obtain multi-scale classification results and target scale classification results, including: (1) A multi-branch feature extraction network is constructed, and a multi-scale feature fusion mechanism is proposed; the multi-branch feature extraction network is composed of multiple parallel convolution branches, each convolution branch processes super-resolution images of different scales; each convolution branch includes several convolution layers, batch normalization layers and ReLU activation function layers, extracts the spatial-spectral features of each scale super-resolution image, and the obtained feature size is 128×1×1; the extracted multi-scale features are fused through the multi-scale feature fusion mechanism to generate multi-scale fused spatial-spectral features; the multi-branch feature extraction and multi-scale feature fusion can be expressed by the following formula: Among them, X L represents the original hyperspectral image, Represents the 2 predicted by the super-resolution network n times the super-resolution image, FE(·) represents the feature extraction network, cat(·) represents the feature fusion mechanism, and Represent the output characteristics of each branch, and Respectively represent 2 N-1 Multiple-scale output features, target-scale output features, and multi-scale fusion features; (2) and After passing through three fully connected layers, the number of input nodes is set to 128 and (N+1)×128 respectively, and the number of output nodes of each layer is set to 128, 64 and C respectively. Then the softmax function is used to obtain the probability of each category, and the multi-scale classification results and target scale classification results are obtained.
7. The hyperspectral image super-resolution and classification joint optimization method based on diffusion model as claimed in claim 1, characterized in that: In step 6, a class-guided super-resolution dynamic refinement mechanism is constructed, and the multi-scale classification results are used to guide the target scale image to learn new knowledge and further reconstruct high-quality images, including: (1) At the reverse denoising time step T, the loss contribution calculation is performed on the obtained multi-scale classification results and the target scale classification results, so that the model pays more attention to inconsistent samples; the loss contribution calculation is expressed by the formula: Among them, M is the total number of categories, Y i,c Indicates whether there are sample points belonging to category c in the multi-scale classification results. represents the probability that the target scale predicts that sample i belongs to class c, γ is the adjustment factor, Indicates the loss contribution consistent with the target scale and multi-scale classification results; (2) Using the following formula, Direct the diffusion sampling process to the desired classification label to further refine the super-resolution results: Among them, s is the classification guidance factor, Represents the last level of the ring multi-layer optimization network Θ in the super-resolution network sN The predicted image, and Respectively represent the input and output of the super-resolution network at time step T, Represented by Θ sN The mean value obtained, σ 2 represents the variance obtained by the forward process, I represents the identity matrix, ω * represents the optimal network parameters of the model, represents a normally distributed process.
8. The hyperspectral image super-resolution and classification joint optimization method based on diffusion model as claimed in claim 1, characterized in that: Step 7: Design a multi-task learning strategy to jointly optimize image super-resolution and classification tasks. The multi-scale super-resolution network, multi-head classification network and class-guided super-resolution dynamic refinement mechanism together constitute a multi-task learning model. Image super-resolution and classification tasks are performed at each time step of reverse denoising to achieve joint optimization of the two tasks, including: (1) Design a multi-task learning strategy that jointly optimizes image super-resolution and classification tasks. Integrate the multi-scale super-resolution network, multi-head classification network, and class-oriented super-resolution dynamic refinement mechanism to build a multi-task learning model. At the time step T of reverse denoising, the multi-scale super-resolution network can be expressed as: Among them, p(X T ) represents the pure noise image X at time step T T It follows a normal distribution with zero mean and unit variance. represents the reverse denoising process, Represented by the multi-scale super-resolution network Θ s The predicted mean, σ 2 represents the variance obtained by the forward process, ω * represents the optimal network parameters of the model; The multi-head classification network can be expressed as: Among them, Y and y n They are respectively represented as multi-scale classification results and target scale classification results, Θ c (·) represents a multi-head classification network, Represents the multi-scale super-resolution network Θ s Predicted super-resolution images at multiple scales; Class-oriented super-resolution dynamic refinement mechanism Θ Guide It can be expressed as: in, and Represent the two states of the target scale image, It is a super-resolution result that needs further refinement; Constructed multi-task learning model Θ M It can be expressed as: (2) In each time step, super-resolution reconstruction and classification tasks of the image are performed simultaneously. The super-resolution task provides more detailed spatial texture information for the classification network, thereby improving the classification performance. At the same time, the classification task uses the class-guided super-resolution dynamic refinement mechanism to provide more detailed optimization directions for the super-resolution network, which is conducive to further refinement of the super-resolution image, thereby achieving joint optimization of the two tasks.
9. The hyperspectral image super-resolution and classification joint optimization method based on diffusion model as claimed in claim 1, characterized in that: In step 7, supervised training is performed on the constructed multi-task learning model to obtain the optimal network parameters of the model, including: (1) The super-resolution task training sample set with reference images Each time, 4 samples are randomly selected as a batch and input into the multi-scale super-resolution network to be trained, and the image prediction of the training samples is output; the labeled classification task training sample set Each time, 16 samples are randomly selected as a batch and input into the multi-head classification network to be trained, and the label prediction of the training samples is output; (2) Use the following loss function formula to calculate the loss function between the predicted image and the reference image, and between the predicted label and the true label of the reference image in the super-resolution and classification tasks respectively: in, is the final loss function, To rebuild the losses, is the classification loss, λ is the weight coefficient, Represents multiple scale resolution images predicted by the super-resolution network, X0 is The corresponding reference image, represents the mean operation, ||·||1 represents the L1-norm, M represents the total number of categories, Indicates whether there are sample points in the reference image label that belong to category c. and i,c They represent the target scale and the probability that the multi-scale predicted sample i belongs to class c respectively; (3) The network parameters are trained using the adaptive moment estimation algorithm Adam until the network converges to the optimal value and the optimal network parameters are saved.
