Spectral and Spatial Aggregation Guided Remote Sensing Hyperspectral Sharpening Fidelity Method and System

Abstract

The present invention relates to the technical field of electrical digital data processing, and in particular to a method and system for remote sensing hyperspectral sharpening and fidelity guided by spectral and spatial aggregation. The method includes the following steps: obtaining hyperspatial pixels, a spatial pixel affinity matrix, hyperspectral channels, and a spectral affinity matrix; performing information interaction to obtain the interaction features of the hyperspatial pixels and the interaction features of the hyperspectral channels; performing self-attention interaction to obtain the updated hyperspectral features of each hyperspectral channel and the updated hyperspectral features; and fusing the updated hyperspatial features with the updated hyperspectral features to obtain a remote sensing hyperspectral sharpened and fidelity image. The method and system provided by the present invention achieve an improvement in spatial resolution while ensuring the integrity of the spectral dimension through spatial and spectral feature aggregation and interaction mechanisms, providing strong technical support for the analysis and practical application of remote sensing hyperspectral images.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing or generation, and particularly to a remote sensing hyperspectral sharpening and fidelity method and system guided by spectral and spatial aggregation. Background Art

[0002] Remote sensing hyperspectral imaging obtains the characteristics of the Earth's surface through many continuous spectral bands. It is widely used in fields such as national land monitoring, agriculture and forestry surveys, disaster warning, and military defense. However, the hyperspectral imaging process has natural difficulties in balancing spatial resolution and spectral resolution. Usually, to achieve a higher spectral resolution, it will inevitably lead to too low a spatial resolution, and the limited spatial resolution will cause spectral mixing of endmembers, affecting the detection performance based on hyperspectral image technology. To obtain high-spatial-resolution hyperspectral images, the technology of pansharpening that fuses low-spatial-resolution hyperspectral images and panchromatic images has emerged.

[0003] Traditional hyperspectral image sharpening and fusion super-resolution algorithms mostly rely on technical frameworks based on matrix decomposition, sparse representation, and deep learning. However, these methods have significant limitations in modeling the complex spectral and spatial characteristics of hyperspectral data: on the one hand, existing methods are difficult to fully capture the similar correlations between multispectral channels, easily ignoring rich spectral information and resulting in spectral information distortion; on the other hand, existing deep learning methods often do not consider the information of the two source images, and have limited ability to mine and interact with information, resulting in the fusion result being difficult to fully utilize both spectral and spatial information. In addition, traditional methods perform poorly in terms of adaptability to multi-sensor and multi-scene data, and it is difficult to meet the actual application requirements. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a remote sensing hyperspectral sharpening and fidelity method and system guided by spectral and spatial aggregation, design a spatial and spectral feature aggregation and interaction mechanism, strive to achieve an improvement in spatial resolution on the premise of ensuring the integrity of the spectral dimension, construct a fusion module with noise immunity characteristics, effectively cope with the interference brought by multi-source data differences, and provide strong technical support for the analysis and practical application of remote sensing hyperspectral images.

[0005] The present invention is realized by the following technical solutions:

[0006] A remote sensing hyperspectral sharpening and fidelity method guided by spectral and spatial aggregation, which includes the following steps:

[0007] S1: Divide the original space of the remotely sensed hyperspectral image in the training set into multiple hyperspace pixel grids, and obtain the corresponding hyperspace pixels and spatial pixel affinity matrix through the way of clustering iteration. Divide the spectral channels of the remotely sensed hyperspectral image in the training set into multiple hyperspectral channel grids, and obtain the corresponding hyperspectral channels and spectral affinity matrix through the way of clustering iteration;

[0008] S2: Perform information interaction between the original space pixels of the remotely sensed hyperspectral image and the hyperspace pixels and hyperspectral channels to obtain the interaction features of the hyperspace pixels. Perform information interaction between the original spectral channels of the remotely sensed hyperspectral image and the hyperspectral channels and hyperspace pixels to obtain the interaction features of the hyperspectral channels;

[0009] S3: Based on the interaction features of the hyperspace pixels, select multiple pixels that are most similar to each hyperspace pixel according to the spatial pixel affinity matrix, and obtain the updated hyperspace features of each hyperspace pixel according to the self-attention mechanism. Based on the interaction features of the hyperspectral channels, select multiple spectral channels that are most similar to each hyperspectral channel according to the spectral affinity matrix, and obtain the updated hyperspectral features of each hyperspectral channel according to the self-attention mechanism;

[0010] S4: Aggregate the updated hyperspace features of all hyperspace pixels to obtain the updated hyperspace features, and aggregate the updated hyperspectral features of all hyperspectral channels to obtain the updated hyperspectral features;

[0011] S5: Fuse the updated hyperspace features and the updated hyperspectral features to obtain the remotely sensed hyperspectral sharpened and fidelity image.

[0012] Optimized, the following method is adopted in step S1 to obtain the hyperspace pixels and spatial pixel affinity matrix:

[0013] S11: Regularly divide the grid of the original space of the remotely sensed hyperspectral image in the training set, and obtain the original single pixel values and the original spatial pixel affinity matrix through sampling;

[0014] S12: Obtain the spatial pixel affinity matrix after a set number of clustering iterations, and weight all the spatial pixels in the spatial pixel affinity matrix to obtain the hyperspace pixels.

[0015] Optimized, the following method is adopted in step S1 to obtain the hyperspectral channels and spectral affinity matrix:

[0016] S13: Regularly divide the grid of the spectral channels of the remotely sensed hyperspectral image in the training set to obtain the original spectral affinity matrix;

[0017] S14: After obtaining the spectral affinity matrix through a set number of clustering iterations, all spectral channels in the spectral affinity matrix are weighted to obtain the hyperspectral channel.

[0018] Further, in step S2, the following method is used to perform information interaction between the original spatial pixels of the remotely sensed hyperspectral image, the hyperspatial pixels, and the hyperspectral channel to obtain the interaction features of the hyperspatial pixels:

[0019] S21: Calculate the query of the hyperspatial pixel, the key of the hyperspatial pixel, and the value of the hyperspatial pixel according to the linear projection, and then use the interactive attention mechanism to calculate the preliminary hyperspatial pixel interaction features after information interaction between the original spatial pixel and the hyperspatial pixel according to formula (1):

[0020] (1);

[0021] Where: represents the preliminary hyperspatial pixel interaction features, represents the normalization exponential function, represents the query of the hyperspatial pixel, represents the key of the hyperspatial pixel, represents the value of the hyperspatial pixel, represents the scaling factor, represents the matrix transpose;

[0022] S22: Re-obtain the query of the hyperspatial pixel from the original spatial pixel features, use the key of the hyperspectral channel as the key, and the preliminary hyperspatial pixel interaction features as the value. Use the interactive attention mechanism to further perform information interaction between the hyperspatial pixel features after preliminary information interaction and the original space to obtain the interaction features of the hyperspatial pixels after information interaction between the original spatial pixel, the hyperspatial pixel, and the hyperspectral channel.

[0023] Further, in step S2, the following method is used to perform information interaction between the original spectral channels of the remotely sensed hyperspectral image, the hyperspectral channel, and the hyperspatial pixels to obtain the interaction features of the hyperspectral channel:

[0024] S23: Calculate the query of the hyperspectral channel, the key of the hyperspectral channel, and the value of the spectral channel according to the linear projection, and then use the interactive attention mechanism to calculate the preliminary hyperspectral channel interaction features after information interaction between the original spectral channel and the hyperspectral channel according to formula (2):

[0025] (2);

[0026] Where: represents the preliminary hyperspectral channel interaction features, represents the normalization exponential function, represents the query of the hyperspectral channel, The key representing the hyperspectral channel The value representing the hyperspectral channel The scaling factor The matrix transpose

[0027] S24: Retrieve the query of the hyperspectral channel from the original spectral channels, use the key of the hyperspatial pixel as the key, and the preliminary hyperspectral channel interaction feature as the value. Then, use the interaction attention mechanism to further interact the hyperspectral channel feature after preliminary information interaction with the original spectral channels, and obtain the interaction feature of the hyperspectral channel that interacts with the original spectral channels, hyperspatial pixels, and hyperspectral channels

[0028] Furthermore, in step S3, the following method is used to obtain the updated hyperspatial feature of each hyperspatial pixel

[0029] S31: Sort the pixels in the spatial pixel affinity matrix in descending order, select the top pixels, and obtain the query, key, and value of the self-attention spatial pixels through linear transformation according to the self-attention mechanism. Then, calculate the attention within the superpixel according to equation (3) to obtain the feature after attention calculation within the hyperspatial pixel

[0030] (3);

[0031] Wherein Represents the feature after attention calculation within the hyperspatial pixel Represents the query of the self-attention spatial pixel Represents the key of the self-attention spatial pixel Represents the value of the self-attention spatial pixel

[0032] S32: Re-perform projection calculation based on the feature after attention calculation within the hyperspatial pixel to obtain the updated hyperspatial feature of each hyperspatial pixel

[0033] Furthermore, in step S3, the following method is used to obtain the updated hyperspectral feature of each hyperspectral channel

[0034] S33: Sort the spectral channels in the spectral affinity matrix in descending order, select the top spectral channels, and obtain the query, key, and value of the self-attention spectral channels through linear transformation according to the self-attention mechanism. Then, calculate the attention within the spectral channel according to equation (4) to obtain the feature after attention calculation within the hyperspectral channel

[0035] (4);

[0036] Wherein: represents the feature after hyperspectral channel internal attention calculation, represents the query of the self-attention spectral channel, represents the key of the self-attention spectral channel, represents the value of the self-attention spectral channel;

[0037] S34: Re-perform projection calculation based on the feature after hyperspectral channel internal attention calculation to obtain the updated hyperspectral feature of each hyperspectral channel.

[0038] Furthermore, in step S32, the feature after hyperspatial pixel internal attention calculation is re-projected according to formula (5) to obtain the updated hyperspatial feature of each hyperspatial pixel:

[0039] (5);

[0040] Wherein: represents the updated hyperspatial feature of each hyperspatial pixel, represents the weight matrix corresponding to the feature before self-attention information interaction within the hyperspatial pixel.

[0041] Furthermore, in step S34, the feature after hyperspectral channel internal attention calculation is re-projected according to formula (6) to obtain the updated hyperspectral feature of each hyperspectral channel:

[0042] (6);

[0043] Wherein: represents the updated hyperspectral feature of each hyperspectral channel, represents the weight matrix corresponding to the feature before self-attention information interaction within the hyperspectral channel.

[0044] A remote sensing hyperspectral sharpening and fidelity system guided by spectral and spatial aggregation is used to execute any of the above-mentioned remote sensing hyperspectral sharpening and fidelity methods guided by spectral and spatial aggregation. It includes a hyperspatial clustering module, a hyperspectral clustering module, a hyperspatial interactive attention module, a hyperspectral interactive attention module, a hyperspatial self-attention module, a hyperspectral self-attention module, a hyperspatial feature aggregation module, a hyperspectral feature aggregation module, and a remote sensing hyperspectral sharpening and fidelity image fusion module;

[0045] The hyperspatial clustering module is used to divide the original space of the remote sensing hyperspectral image in the training set into multiple hyperspatial pixel grids, and obtain the corresponding hyperspatial pixels and spatial pixel affinity matrix through clustering iteration;

[0046] The hyperspectral clustering module is used to divide the spectral channels of remote sensing hyperspectral images in the training set into multiple hyperspectral channel grids, and obtain the corresponding hyperspectral channels and spectral affinity matrix through clustering iteration;

[0047] The hyperspatial interaction attention module is used to perform information interaction between the original spatial pixels of the remote sensing hyperspectral image, hyperspatial pixels, and hyperspectral channels to obtain the interaction features of the hyperspatial pixels;

[0048] The hyperspectral self-attention module is used to perform information interaction between the original spectral channels of the remote sensing hyperspectral image, hyperspectral channels, and hyperspatial pixels to obtain the interaction features of the hyperspectral channels;

[0049] The hyperspatial feature aggregation module is used to aggregate the updated hyperspatial features of all hyperspatial pixels to obtain the updated hyperspatial features;

[0050] The hyperspectral feature aggregation module is used to aggregate the updated hyperspectral features of all hyperspectral channels to obtain the updated hyperspectral features;

[0051] The remote sensing hyperspectral sharpening and fidelity image fusion module is used to fuse the updated hyperspatial features and the updated hyperspectral features to obtain the remote sensing hyperspectral sharpening and fidelity image.

[0052] Advantages of the invention:

[0053] The spectral and spatial aggregation-guided remote sensing hyperspectral sharpening and fidelity method and system provided by the present invention have the following advantages:

[0054] (1) By adopting the methods of spectral clustering iteration and spatial clustering iteration, the characteristic information of the similar regions inside the spectrum and the characteristic information of the similar regions inside the spatial pixels are effectively obtained, and the information of the spectral and spatial similar regions is fully utilized.

[0055] (2) On the basis of adopting clustering of similar spectral and spatial regions, an interactive attention mechanism is used to form hyperspectral channels and hyperspatial pixels, which promotes the interaction of information in distant similar regions and enhances the information utilization efficiency.

[0056] (3) On the basis of the self-attention mechanism, the most relevant characteristic information is selected, which ensures the efficient concentration of information interaction and eliminates interference and redundant calculations.

[0057] (4) By performing double-branch interactive guidance, the spatial information and spectral information of the panchromatic image and hyperspectral image data can be utilized more effectively, and better spectral and spatial fidelity effects can be achieved. Description of the drawings

[0058] Figure 1 is a schematic diagram of the process of the present invention.

[0059] Figure 2 It is a schematic diagram of the system structure of the present invention. Detailed implementation manner

[0060] A remote sensing hyperspectral sharpening and fidelity method guided by spectral and spatial aggregation, which includes the following steps, and its flowchart is as Figure 1 shown:

[0061] S1: Divide the original space of the remote sensing hyperspectral image in the training set into multiple hyperspace pixel grids, and obtain the corresponding hyperspace pixels and spatial pixel affinity matrices through the method of clustering iteration. Divide the spectral channels of the remote sensing hyperspectral image in the training set into multiple hyperspectral channel grids, and obtain the corresponding hyperspectral channels and spectral affinity matrices through the method of clustering iteration;

[0062] Specifically, the following method can be used to obtain hyperspace pixels and spatial pixel affinity matrices:

[0063] S11: Regularly divide the grid of the original space of the remote sensing hyperspectral image in the training set, and obtain the original single-pixel values and the original spatial pixel affinity matrix through sampling;

[0064] S12: After a set number of clustering iterations, obtain the spatial pixel affinity matrix, and weight all the spatial pixels in the spatial pixel affinity matrix to obtain hyperspace pixels.

[0065] Assume that the size of the originally divided grid is , then the number of pixels of the hyperspace pixels that can be calculated is , after multiple clustering iterations, the spatial affinity matrix can be obtained;

[0066] Where: represents the height of the grid divided by the original space of the remote sensing hyperspectral image, represents the width of the grid divided by the original space of the remote sensing hyperspectral image, represents the number of hyperspace pixels, represents the height of the remote sensing hyperspectral image, represents the width of the remote sensing hyperspectral image, represents the th row and the th column of spatial pixels, represents the th row of pixels, , represents the real vector space with the dimension of the number of spectral channels, represents after iterations, the th column of hyperspace pixels, Denotes the square of the two - norm, Denotes the natural constant.

[0067] The hyperspace pixel can be obtained by weighting the spatial pixels, as shown in the following formula:

[0068] ;

[0069] Where: Denotes The hyperspace pixel of the th column after the Denotes the normalization of the spatial pixel along the column, .

[0070] Specifically, the following method can be used to obtain the hyperspectral channel and the spectral affinity matrix:

[0071] S13: Regularly divide the grid of the spectral channels of the remote - sensing hyperspectral image in the training set to obtain the original spectral affinity matrix;

[0072] S14: After a set number of clustering iterations, obtain the spectral affinity matrix, and weight all the spectral channels in the spectral affinity matrix to obtain the hyperspectral channel.

[0073] Assume that the size of the divided hyperspectral channel grid is , so that the number of hyperspectral channels can be calculated as , where Denotes the number of hyperspectral channels, Denotes the total number of spectral channels of the remote - sensing hyperspectral image; after times of clustering iterations, the spectral affinity matrix and the final hyperspectral channel can also be obtained:

[0074] ;

[0075] ;

[0076] Where: Denotes the spectral affinity matrix, Denotes the number of spectral channels in the th row, Denotes The hyperspectral channel of the th column after the Denotes The hyperspectral channel of the th column after the Denotes the normalization of the spectral channel along the column.

[0077] Different from the conventional method of regularly dividing the input image into blocks, in this application, the image is divided into individual superpixels. Through continuous clustering iterations, a spatial pixel affinity matrix and hyperspatial pixels are obtained, which can take into account the similarity between pixels and connect them together. In this way, compared with the regular segmentation method, it can reconstruct more accurate boundaries and reduce boundary blurring. Moreover, during the spatial aggregation process, only the correlation between each spatial pixel and its surrounding superpixels is calculated, which ensures the locality of the hyperspatial pixels and enables high computational efficiency in terms of both calculation and memory. Applying the idea of superpixel clustering to the spectral channels can achieve the same effect in the spectral channels.

[0078] S2: Perform information interaction between the original spatial pixels of the remote sensing hyperspectral image, the hyperspatial pixels, and the hyperspectral channels to obtain the interaction features of the hyperspatial pixels. Perform information interaction between the original spectral channels of the remote sensing hyperspectral image, the hyperspectral channels, and the hyperspatial pixels to obtain the interaction features of the hyperspectral channels;

[0079] Furthermore, in step S2, the following method is used to perform information interaction between the original spatial pixels of the remote sensing hyperspectral image, the hyperspatial pixels, and the hyperspectral channels to obtain the interaction features of the hyperspatial pixels:

[0080] S21: Calculate the query of the hyperspatial pixels, the key of the hyperspatial pixels, and the value of the hyperspatial pixels according to linear projection. Then, use the interactive attention mechanism to calculate the preliminary hyperspatial pixel interaction features after information interaction between the original spatial pixels and the hyperspatial pixels according to Equation (1):

[0081] (1);

[0082] Where: represents the preliminary hyperspatial pixel interaction features, represents the normalization exponential function, represents the query of the hyperspatial pixels, represents the key of the hyperspatial pixels, represents the value of the hyperspatial pixels, represents the scaling factor, represents the matrix transpose;

[0083] Here, the query of the hyperspatial pixels , , where represents the real vector space with the dimension of the spatial dimension multiplied by the number of spectral channels, represents the given hyperspatial pixel features, represents the weight matrix corresponding to the query of the hyperspatial pixels; the key of the hyperspatial pixels , , represents a real vector space with dimensions of the height times the width times the number of spectral channels of the remote sensing hyperspectral image, represents the original space pixel features, represents the weight matrix corresponding to the key of the hyperspace pixel; the value of the hyperspace pixel , , represents the weight matrix corresponding to the value of the hyperspace pixel;

[0084] S22: Retrieve the query of the hyperspace pixel from the original space pixel features, use the keys of the hyperspectral channels as keys, and the preliminary hyperspace pixel interaction features as values. Utilize the interactive attention mechanism to further interact the hyperspace pixel features after preliminary information interaction with the original space, obtaining the interaction features of the hyperspace pixel after information interaction between the original space pixels, the hyperspace pixels, and the hyperspectral channels.

[0085] The difference between this step and the hyperspace pixel clustering in step S1 is that this process is not restricted by the neighborhood. The obtained interaction features of the hyperspace pixel are the long-distance interaction information of the space pixels, realizing the long-distance information interaction of the space pixels.

[0086] Furthermore, in step S2, the following method is adopted to interact the original spectral channels of the remote sensing hyperspectral image with the hyperspectral channels and the hyperspace pixels to obtain the interaction features of the hyperspectral channels:

[0087] S23: Calculate the query of the hyperspectral channel, the key of the hyperspectral channel, and the value of the spectral channel according to the linear projection, and then use the interactive attention mechanism to calculate the preliminary hyperspectral channel interaction features after information interaction between the original spectral channel and the hyperspectral channel according to equation (2):

[0088] (2);

[0089] Where: represents the preliminary hyperspectral channel interaction features, represents the normalization exponential function, represents the query of the hyperspectral channel, represents the key of the hyperspectral channel, represents the value of the hyperspectral channel, represents the scaling factor, represents the matrix transpose;

[0090] Here, the query of the hyperspectral channel , , represents a real vector space with dimensions of the height times the width times the spatial dimension of the remote sensing hyperspectral image, represents the number of hyperspectral channels, Represents the given hyperspectral channel feature, Represents the weight matrix corresponding to the query of the hyperspectral channel; the key of the hyperspectral channel , , Represents a real vector space with dimensions of the height times the width times the number of spectral channels of the remote sensing hyperspectral image, Represents the spectral channel, , Represents the weight matrix corresponding to the key of the hyperspectral channel; the value of the hyperspectral channel , , Represents the weight matrix corresponding to the value of the hyperspectral channel;

[0091] S24: Retrieve the query of the hyperspectral channel from the original spectral channels, use the key of the hyperspatial pixel as the key, and the preliminary hyperspectral channel interaction feature as the value. Utilize the interactive attention mechanism to further interact the hyperspectral channel feature after the preliminary information interaction with the original spectral channels, and obtain the interaction feature of the hyperspectral channel that interacts with the original spectral channels, hyperspectral channels, and hyperspatial pixels.

[0092] The difference between this step and the hyperspectral channel aggregation in step S1 is that this process is not restricted by the neighborhood. The obtained interaction feature of the hyperspectral channel is the long-distance interaction information of the spectral channels, realizing the long-distance information interaction of the spectral channels.

[0093] Through the clustering operations of hyperspace and hyperspectral in step S1, we can capture the local correlations of spatial pixels and spectral channels in the region. However, for the hyperspectral pan-sharpening task, this may lack the ability to capture the long-range dependencies of spatial pixels and spectral channels. Therefore, we borrow the idea of the attention mechanism to enhance the interaction of spatial pixels and spectral channels for long-range information, which can greatly utilize the information correlation between features and generate higher-quality images. Because there is a great correlation between spatial pixels or spectral channels and their corresponding hyperspatial pixels or hyperspectral channels, using interactive attention can make full use of pixel or spectral information as much as possible, ensuring the long-distance information interaction and enhancing the information utilization efficiency.

[0094] S3: Based on the interaction features of hyperspatial pixels, select multiple pixels that are most similar to each hyperspatial pixel according to the spatial pixel affinity matrix, and obtain the updated hyperspatial features of each hyperspatial pixel according to the self-attention mechanism. Based on the interaction features of hyperspectral channels, select multiple spectral channels that are most similar to each hyperspectral channel according to the spectral affinity matrix, and obtain the updated hyperspectral features of each hyperspectral channel according to the self-attention mechanism;

[0095] Specifically, the following method can be used to obtain the updated hyperspace features of each hyperspace pixel:

[0096] S31: Sort the pixels in the spatial pixel affinity matrix in descending order, and select the top pixels. According to the self-attention mechanism, obtain the query of the self-attention spatial pixel, the key of the self-attention spatial pixel, and the value of the self-attention spatial pixel through linear transformation. Then, perform attention calculation on the inside of the superpixel according to Equation (3) to obtain the feature after attention calculation inside the hyperspace pixel:

[0097] (3);

[0098] Where: represents the feature after attention calculation inside the hyperspace pixel, represents the query of the self-attention spatial pixel, , represents the weight matrix corresponding to the query of the self-attention spatial pixel, represents the key of the self-attention spatial pixel, , represents the weight matrix corresponding to the key of the self-attention spatial pixel, represents the value of the self-attention spatial pixel, , represents the weight matrix corresponding to the value of the self-attention spatial pixel;

[0099] S32: Based on the feature after attention calculation inside the hyperspace pixel, perform projection calculation again to obtain the updated hyperspace features of each hyperspace pixel.

[0100] Specifically, the feature based on the attention calculation inside the hyperspace pixel can be re-projected according to Equation (5) to obtain the updated hyperspace features of each hyperspace pixel:

[0101] (5);

[0102] Where: represents the updated hyperspace features of each hyperspace pixel, represents the weight matrix corresponding to the feature before self-attention information interaction inside the hyperspace pixel.

[0103] Furthermore, in step S3, the following method is used to obtain the updated hyperspectral features of each hyperspectral channel:

[0104] S33: Sort the spectral channels in the spectral affinity matrix in descending order, and select the top The spectral channels obtain the query of the self-attention spectral channels, the key of the self-attention spectral channels, and the value of the self-attention spectral channels through linear transformation according to the self-attention mechanism, and then perform attention calculation on the inside of the spectral channels according to Equation (4) to obtain the features after the internal attention calculation of the hyperspectral channels:

[0105] (4);

[0106] Where: represents the features after the internal attention calculation of the hyperspectral channels, represents the query of the self-attention spectral channels, , represents the weight matrix corresponding to the query of the internal attention spectral channels, represents the key of the self-attention spectral channels, , represents the weight matrix corresponding to the key of the internal attention spectral channels, represents the value of the self-attention spectral channels, , represents the weight matrix corresponding to the value of the internal attention spectral channels;

[0107] S34: Re-perform projection calculation based on the features after the internal attention calculation of the hyperspectral channels to obtain the updated hyperspectral features of each hyperspectral channel.

[0108] Furthermore, in step S34, the features after the internal attention calculation of the hyperspectral channels are re-projected according to Equation (6) to obtain the updated hyperspectral features of each hyperspectral channel:

[0109] (6);

[0110] Where: represents the updated hyperspectral features of each hyperspectral channel, represents the weight matrix corresponding to the features before the self-attention information interaction within the hyperspectral channels.

[0111] S4: Aggregate the updated hyperspatial features of all hyperspatial pixels to obtain the updated hyperspatial features, and aggregate the updated hyperspectral features of all hyperspectral channels to obtain the updated hyperspectral features;

[0112] Since different hyperspatial pixels or hyperspectral channels have different numbers of affiliated spatial pixels or spectral channels, some will have a large number of spatial pixels or spectral channels. This will affect the utilization of complementary information of similar pixels or channels within the hyperspatial pixels or hyperspectral channels, and will cause additional computational and storage consumption.

[0113] Therefore, the present invention uses the above method for self-attention processing, which can select the most relevant feature information, thereby prompting the network to focus more on effective information, ensuring the efficient concentration of information interaction, and eliminating interference and redundant calculations.

[0114] S5: Fuse the updated hyperspace features and the updated hyperspectral features to obtain a remote sensing hyperspectral sharpened and fidelity image.

[0115] By constructing a double-branch interaction-guided network, the present invention can make more effective use of the spatial information and spectral information of the panchromatic image and the hyperspectral image data, thereby achieving better spectral and spatial fidelity effects.

[0116] To verify that the present invention has better effects, we conducted a comparative experiment on quantitative index results with a variety of currently most advanced international algorithms on the public dataset Pavia Center. The experimental results are shown in Table 1:

[0117] Table 1:

[0118]

[0119] As can be seen from Table 1, all the quantitative indexes of the present invention are better than those of the internationally advanced algorithms in recent years.

[0120] A remote sensing hyperspectral sharpened and fidelity system guided by spectral and spatial aggregation is used to execute the remote sensing hyperspectral sharpened and fidelity method guided by spectral and spatial aggregation as described in any one of the above. The schematic diagram of the system structure is as Figure 2 shown, which includes a hyperspace clustering module, a hyperspectral clustering module, a hyperspace interaction attention module, a hyperspectral interaction attention module, a hyperspace self-attention module, a hyperspectral self-attention module, a hyperspace feature aggregation module, a hyperspectral feature aggregation module, and a remote sensing hyperspectral sharpened and fidelity image fusion module;

[0121] The hyperspace clustering module is used to divide the original space of the remote sensing hyperspectral image in the training set into multiple hyperspace pixel grids, and obtain the corresponding hyperspace pixels and spatial pixel affinity matrix through clustering iteration;

[0122] The hyperspectral clustering module is used to divide the spectral channels of the remote sensing hyperspectral image in the training set into multiple hyperspectral channel grids, and obtain the corresponding hyperspectral channels and spectral affinity matrix through clustering iteration;

[0123] The hyperspace interaction attention module is used to perform information interaction between the original spatial pixels of the remote sensing hyperspectral image, the hyperspace pixels, and the hyperspectral channels to obtain the interaction features of the hyperspace pixels;

[0124] The hyperspectral self-attention module is used to interact information between the original spectral channels, hyperspectral channels, and hyperspatial pixels of remote sensing hyperspectral images to obtain the interaction features of the hyperspectral channels;

[0125] The hyperspatial feature aggregation module is used to aggregate the updated hyperspatial features of all hyperspatial pixels to obtain the updated hyperspatial features;

[0126] The hyperspectral feature aggregation module is used to aggregate the updated hyperspectral features of all hyperspectral channels to obtain the updated hyperspectral features;

[0127] The remote sensing hyperspectral sharpening and fidelity image fusion module is used to fuse the updated hyperspatial features and the updated hyperspectral features to obtain the remote sensing hyperspectral sharpening and fidelity image.

[0128] In summary, the spectral and spatial aggregation-guided remote sensing hyperspectral sharpening and fidelity method and system provided by the present invention achieve the improvement of spatial resolution while ensuring the integrity of the spectral dimension by designing the spatial and spectral feature aggregation and interaction mechanisms, and construct a fusion module with noise immunity characteristics, which can effectively cope with the interference caused by the differences in multi-source data, providing strong technical support for the analysis and practical application of remote sensing hyperspectral images.

[0129] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A remote sensing hyperspectral sharpening and fidelity preservation method guided by spectral and spatial aggregation, characterized by: The steps include: S1: Divide the original space of the remote sensing hyperspectral image in the training set into multiple hyperspatial pixel grids, and obtain the corresponding hyperspatial pixels and spatial pixel affinity matrices through clustering iteration. Divide the spectral channels of the remote sensing hyperspectral image in the training set into multiple hyperspectral channel grids, and obtain the corresponding hyperspectral channels and spectral affinity matrices through clustering iteration. S2: The original spatial pixels of the remote sensing hyperspectral image are interacted with the hyperspatial pixels and the hyperspectral channels to obtain the interactive features of the hyperspatial pixels; the original spectral channels of the remote sensing hyperspectral image are interacted with the hyperspectral channels and the hyperspatial pixels to obtain the interactive features of the hyperspectral channels; S3: Based on the interactive features of hyperspace pixels, multiple pixels that are most similar to each hyperspace pixel are selected according to the spatial pixel affinity matrix, and the updated hyperspace features of each hyperspace pixel are obtained according to the self-attention mechanism. Based on the interactive features of hyperspectral channels, multiple spectral channels that are most similar to each hyperspectral channel are selected according to the spectral affinity matrix, and the updated hyperspectral features of each hyperspectral channel are obtained according to the self-attention mechanism. S4: Aggregate the updated hyperspace features of all hyperspace pixels to obtain updated hyperspace features, and aggregate the updated hyperspectral features of all hyperspectral channels to obtain updated hyperspectral features; S5: Fuse the updated hyperspatial features with the updated hyperspectral features to obtain a remote sensing hyperspectral sharpened fidelity image.

2. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 1, characterized in that: In step S1, the following method is used to obtain the hyperspatial pixel and spatial pixel affinity matrix: S11: The original space of the remote sensing hyperspectral image in the training set is divided into regular grids, and the original single pixel value and the original space pixel affinity matrix are obtained by sampling; S12: After a set number of clustering iterations, a spatial pixel affinity matrix is ​​obtained, and all spatial pixels in the spatial pixel affinity matrix are weighted to obtain hyper-spatial pixels.

3. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 1, characterized in that: In step S1, the following method is used to obtain the hyperspectral channel and spectral affinity matrix: S13: dividing the spectral channels of the remote sensing hyperspectral images in the training set into grids regularly to obtain the original spectral affinity matrix; S14: After a set number of clustering iterations, a spectral affinity matrix is ​​obtained, and all spectral channels in the spectral affinity matrix are weighted to obtain a hyperspectral channel.

4. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 1, characterized in that: In step S2, the original spatial pixels of the remote sensing hyperspectral image are interacted with the hyperspatial pixels and the hyperspectral channels by the following method to obtain the interactive features of the hyperspatial pixels: S2 1: Based on the linear projection, the query, key and value of the hyperspace pixel are calculated. Then, the interactive attention mechanism is used to calculate the preliminary hyperspace pixel interaction features after the original space pixel and the hyperspace pixel interact with each other according to formula (1): （1）； in: represents the preliminary hyperspatial pixel interaction features, represents the normalized exponential function, represents a query of a hyperspace pixel, represents the key of the hyperspace pixel, represents the value of the hyperspace pixel, represents the scaling factor, Represents matrix transpose; S22: Re-acquire the query of the hyperspace pixel from the original space pixel feature, and use the key of the hyperspectral channel as the key and the preliminary hyperspace pixel interaction feature as the value, and use the interactive attention mechanism to further interact the hyperspace pixel features after the preliminary information interaction with the original space, so as to obtain the interactive features of the hyperspace pixel after the original space pixel interacts with the hyperspace pixel and the hyperspectral channel.

5. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 1, characterized in that: In step S2, the original spectral channel of the remote sensing hyperspectral image is interacted with the hyperspectral channel and the hyperspatial pixel by the following method to obtain the interactive features of the hyperspectral channel: S23: The query of the hyperspectral channel, the key of the hyperspectral channel and the value of the spectral channel are calculated according to the linear projection, and then the interactive attention mechanism is used to calculate the preliminary hyperspectral channel interaction features after the original spectral channel and the hyperspectral channel interact with each other according to formula (2): （2）； in: represents the preliminary hyperspectral channel interaction characteristics, represents the normalized exponential function, represents the query of hyperspectral channels, represents the key of the hyperspectral channel, represents the value of the hyperspectral channel, represents the scaling factor, Represents matrix transpose; S24: Re-acquire the query of the hyperspectral channel from the original spectral channel, and use the key of the hyperspace pixel as the key and the preliminary hyperspectral channel interaction feature as the value, and use the interactive attention mechanism to further interact the hyperspectral channel features after the preliminary information interaction with the original spectral channel to obtain the interactive features of the hyperspectral channel in which the original spectral channel interacts with the hyperspectral channel and the hyperspace pixel.

6. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 4, characterized in that: In step S3, the updated hyperspace feature of each hyperspace pixel is obtained by the following method: S31: Sort the pixels in the spatial pixel affinity matrix in descending order, and select the first According to the self-attention mechanism, each pixel is linearly transformed to obtain the query of the self-attention space pixel, the key of the self-attention space pixel, and the value of the self-attention space pixel. Then, according to formula (3), the attention calculation is performed on the internal part of the superpixel to obtain the feature after the attention calculation of the internal part of the superspace pixel: （3）； in: represents the features after attention calculation inside the hyperspace pixel, represents the query of pixels in the self-attention space, represents the key of the self-attention space pixel, Represents the value of the pixel in the self-attention space; S32: Re-projection calculation is performed based on the features calculated based on the internal attention of the hyperspace pixel to obtain the updated hyperspace features of each hyperspace pixel.

7. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 5, characterized in that: In step S3, the updated hyperspectral features of each hyperspectral channel are obtained by the following method: S33: Sort the spectral channels in the spectral affinity matrix in descending order, and select the first According to the self-attention mechanism, the spectral channel is linearly transformed to obtain the query of the self-attention spectral channel, the key of the self-attention spectral channel, and the value of the self-attention spectral channel. Then, according to formula (4), the attention calculation is performed inside the spectral channel to obtain the features after the attention calculation inside the hyper-spectral channel: （4）； in: represents the features after attention calculation inside the hyperspectral channel, represents the query of the self-attention spectral channel, represents the key of the self-attention spectral channel, represents the value of the self-attention spectral channel; S34: Re-projection calculation is performed based on the features calculated after the internal attention of the hyperspectral channel to obtain the updated hyperspectral features of each hyperspectral channel.

8. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 6, characterized in that: In step S32, the features calculated based on the internal attention of the hyperspace pixel are reprojected according to formula (5) to obtain the updated hyperspace features of each hyperspace pixel: （5）； in: represents the updated hyperspace feature of each hyperspace pixel, Represents the weight matrix corresponding to the features before the self-attention information interaction in the hyperspace pixel.

9. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 7, characterized in that: In step S34, the features calculated based on the internal attention of the hyperspectral channel are reprojected according to formula (6) to obtain the updated hyperspectral features of each hyperspectral channel: （6）； in: represents the updated hyperspectral features of each hyperspectral channel, Represents the weight matrix corresponding to the features before self-attention information interaction in the hyperspectral channel.

10. A remote sensing hyperspectral sharpening and fidelity preservation system guided by spectral and spatial aggregation, used to execute a remote sensing hyperspectral sharpening and fidelity preservation method guided by spectral and spatial aggregation as claimed in any one of claims 1 to 9, characterized in that: Including hyperspatial clustering module, hyperspectral clustering module, hyperspatial interactive attention module, hyperspectral interactive attention module, hyperspatial self-attention module, hyperspectral self-attention module, hyperspatial feature aggregation module, hyperspectral feature aggregation module and remote sensing hyperspectral sharpening and fidelity image fusion module; The hyperspace clustering module is used to divide the original space of the remote sensing hyperspectral image in the training set into multiple hyperspace pixel grids, and obtain the corresponding hyperspace pixel and space pixel affinity matrix through clustering iteration; The hyperspectral clustering module is used to divide the spectral channels of the remote sensing hyperspectral images in the training set into a plurality of hyperspectral channel grids, and obtain the corresponding hyperspectral channels and spectral affinity matrix through clustering iteration; The hyperspace interactive attention module is used to interact information between the original spatial pixels of the remote sensing hyperspectral image and the hyperspace pixels and hyperspectral channels to obtain interactive features of the hyperspace pixels; The hyperspectral self-attention module is used to perform information interaction between the original spectral channel of the remote sensing hyperspectral image and the hyperspectral channel and hyperspatial pixels to obtain the interactive features of the hyperspectral channel; The hyperspace feature aggregation module is used to aggregate the updated hyperspace features of all hyperspace pixels to obtain updated hyperspace features; The hyperspectral feature aggregation module is used to aggregate the updated hyperspectral features of all hyperspectral channels to obtain updated hyperspectral features; The remote sensing hyperspectral sharpening and fidelity image fusion module is used to fuse the updated hyperspatial features with the updated hyperspectral features to obtain a remote sensing hyperspectral sharpening and fidelity image.

Images