Spectrum and space aggregation guided remote sensing hyperspectral sharpening fidelity method and system

Abstract

The invention relates to the technical field of electric digital data processing, in particular to a spectrum and space aggregation guided remote sensing hyperspectral sharpening fidelity method and system, and the method comprises the following steps: obtaining a hyperspatial pixel, a space pixel affinity matrix, a hyperspectral channel and a spectral affinity matrix; carrying out information interaction to obtain the interaction characteristics of the hyper-space pixels and the interaction characteristics of the hyper-spectral channels; performing self-attention interaction to obtain the updated hyper-spectral features of each hyper-spectral channel and the updated hyper-spectral features of each hyper-spectral channel; and fusing the updated hyper-spatial features and the updated hyper-spectral features to obtain a remote sensing hyperspectral sharpening fidelity image. According to the method and the system provided by the invention, through a space and spectral feature aggregation and interaction mechanism, the spatial resolution is improved on the premise of ensuring the spectral dimension integrity, and a powerful technical support is provided for analysis and practical application of a remote sensing hyperspectral image.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing or generation, and in particular to a remote sensing hyperspectral sharpening and fidelity preservation method and system guided by spectral and spatial aggregation. Background Art

[0002] Remote sensing hyperspectral imaging is to obtain the characteristics of the earth's surface through many continuous spectral bands. It is widely used in land monitoring, agricultural and forestry surveys, disaster warning, military defense and other fields. However, the hyperspectral imaging process has natural difficulties and it is difficult to balance spatial resolution and spectral resolution. Usually, in order to achieve a higher spectral resolution, it is bound to lead to too low spatial resolution, and the limited spatial resolution will lead to spectral mixing of end members, affecting the detection performance based on hyperspectral imaging technology. In order to obtain high spatial resolution hyperspectral images, the technology of fusing low spatial resolution hyperspectral images with panchromatic sharpening of panchromatic images came into being.

[0003] Traditional hyperspectral image sharpening and fusion super-resolution algorithms mostly rely on technical frameworks based on matrix decomposition, sparse representation and deep learning. However, these methods have significant limitations in modeling the complex spectral and spatial characteristics of hyperspectral data: on the one hand, existing methods are difficult to fully capture the similarity correlation between multi-spectral channels, and are prone to ignore rich spectral information, resulting in distortion of spectral information; on the other hand, existing deep learning methods often do not consider the information of the two data source images, and have limited information mining and interaction capabilities, resulting in fusion results that are difficult to ensure full utilization of spectral and spatial information at the same time. In addition, traditional methods perform poorly in terms of adaptability to multi-sensor and multi-scene data, and are difficult to meet the needs of actual applications. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a remote sensing hyperspectral sharpening and fidelity preservation method and system guided by spectral and spatial aggregation, design spatial and spectral feature aggregation and interaction mechanism, and strive to achieve spatial resolution improvement while ensuring the integrity of spectral dimension, construct a fusion module with noise immunity characteristics, effectively deal with the interference caused by differences in multi-source data, and provide strong technical support for the analysis and practical application of remote sensing hyperspectral images.

[0005] The present invention is achieved through the following technical solutions: A remote sensing hyperspectral sharpening and fidelity preservation method guided by spectral and spatial aggregation comprises the following steps: S1: Divide the original space of the remote sensing hyperspectral image in the training set into multiple hyperspatial pixel grids, and obtain the corresponding hyperspatial pixels and spatial pixel affinity matrices through clustering iteration. Divide the spectral channels of the remote sensing hyperspectral image in the training set into multiple hyperspectral channel grids, and obtain the corresponding hyperspectral channels and spectral affinity matrices through clustering iteration. S2: The original spatial pixels of the remote sensing hyperspectral image are interacted with the hyperspatial pixels and the hyperspectral channels to obtain the interactive features of the hyperspatial pixels; the original spectral channels of the remote sensing hyperspectral image are interacted with the hyperspectral channels and the hyperspatial pixels to obtain the interactive features of the hyperspectral channels; S3: Based on the interactive features of hyperspace pixels, multiple pixels that are most similar to each hyperspace pixel are selected according to the spatial pixel affinity matrix, and the updated hyperspace features of each hyperspace pixel are obtained according to the self-attention mechanism. Based on the interactive features of hyperspectral channels, multiple spectral channels that are most similar to each hyperspectral channel are selected according to the spectral affinity matrix, and the updated hyperspectral features of each hyperspectral channel are obtained according to the self-attention mechanism. S4: Aggregate the updated hyperspace features of all hyperspace pixels to obtain updated hyperspace features, and aggregate the updated hyperspectral features of all hyperspectral channels to obtain updated hyperspectral features; S5: Fuse the updated hyperspatial features with the updated hyperspectral features to obtain a remote sensing hyperspectral sharpened fidelity image.

[0006] Optimally, in step S1, the following method is used to obtain the hyperspatial pixel and spatial pixel affinity matrix: S11: The original space of the remote sensing hyperspectral image in the training set is divided into regular grids, and the original single pixel value and the original space pixel affinity matrix are obtained by sampling; S12: After a set number of clustering iterations, a spatial pixel affinity matrix is ​​obtained, and all spatial pixels in the spatial pixel affinity matrix are weighted to obtain hyper-spatial pixels.

[0007] Optimally, in step S1, the following method is used to obtain the hyperspectral channel and spectral affinity matrix: S13: dividing the spectral channels of the remote sensing hyperspectral images in the training set into grids regularly to obtain the original spectral affinity matrix; S14: After a set number of clustering iterations, a spectral affinity matrix is ​​obtained, and all spectral channels in the spectral affinity matrix are weighted to obtain a hyperspectral channel.

[0008] Furthermore, in step S2, the original spatial pixels of the remote sensing hyperspectral image are interacted with the hyperspatial pixels and the hyperspectral channels by the following method to obtain the interactive features of the hyperspatial pixels: S21: The query, key and value of the hyperspace pixel are calculated based on the linear projection. Then, the interactive attention mechanism is used to calculate the preliminary hyperspace pixel interaction features after the original space pixel and the hyperspace pixel interact with each other according to formula (1): (1); in: represents the preliminary hyperspatial pixel interaction features, represents the normalized exponential function, represents a query of a hyperspace pixel, represents the key of the hyperspace pixel, represents the value of the hyperspace pixel, represents the scaling factor, Represents matrix transpose; S22: Re-acquire the query of the hyperspace pixel from the original space pixel feature, and use the key of the hyperspectral channel as the key and the preliminary hyperspace pixel interaction feature as the value, and use the interactive attention mechanism to further interact the hyperspace pixel features after the preliminary information interaction with the original space, so as to obtain the interactive features of the hyperspace pixel after the original space pixel interacts with the hyperspace pixel and the hyperspectral channel.

[0009] Furthermore, in step S2, the original spectral channel of the remote sensing hyperspectral image is interacted with the hyperspectral channel and the hyperspatial pixel by the following method to obtain the interactive features of the hyperspectral channel: S23: The query of the hyperspectral channel, the key of the hyperspectral channel and the value of the spectral channel are calculated according to the linear projection, and then the interactive attention mechanism is used to calculate the preliminary hyperspectral channel interaction features after the original spectral channel and the hyperspectral channel interact with each other according to formula (2): (2); in: represents the preliminary hyperspectral channel interaction characteristics, represents the normalized exponential function, represents the query of hyperspectral channels, represents the key of the hyperspectral channel, represents the value of the hyperspectral channel, represents the scaling factor, Represents matrix transpose; S24: Re-acquire the query of the hyperspectral channel from the original spectral channel, and use the key of the hyperspace pixel as the key and the preliminary hyperspectral channel interaction feature as the value, and use the interactive attention mechanism to further interact the hyperspectral channel features after the preliminary information interaction with the original spectral channel to obtain the interactive features of the hyperspectral channel in which the original spectral channel interacts with the hyperspectral channel and the hyperspace pixel.

[0010] Further, in step S3, the updated hyperspace feature of each hyperspace pixel is obtained by the following method: S31: Sort the pixels in the spatial pixel affinity matrix in descending order, and select the first According to the self-attention mechanism, each pixel is linearly transformed to obtain the query of the self-attention space pixel, the key of the self-attention space pixel, and the value of the self-attention space pixel. Then, according to formula (3), the attention calculation is performed on the internal part of the superpixel to obtain the feature after the attention calculation of the internal part of the superspace pixel: (3); in: represents the features after attention calculation inside the hyperspace pixel, represents the query of pixels in the self-attention space, represents the key of the self-attention space pixel, Represents the value of the pixel in the self-attention space; S32: Re-projection calculation is performed based on the features calculated based on the internal attention of the hyperspace pixel to obtain the updated hyperspace features of each hyperspace pixel.

[0011] Further, in step S3, the updated hyperspectral features of each hyperspectral channel are obtained by the following method: S33: Sort the spectral channels in the spectral affinity matrix in descending order, and select the first According to the self-attention mechanism, the spectral channel is linearly transformed to obtain the query of the self-attention spectral channel, the key of the self-attention spectral channel, and the value of the self-attention spectral channel. Then, according to formula (4), the attention calculation is performed inside the spectral channel to obtain the features after the attention calculation inside the hyper-spectral channel: (4); in: represents the features after attention calculation inside the hyperspectral channel, represents the query of the self-attention spectral channel, represents the key of the self-attention spectral channel, represents the value of the self-attention spectral channel; S34: Re-projection calculation is performed based on the features calculated after the internal attention of the hyperspectral channel to obtain the updated hyperspectral features of each hyperspectral channel.

[0012] Furthermore, in step S32, the features calculated based on the internal attention of the hyperspace pixel are reprojected according to formula (5) to obtain the updated hyperspace features of each hyperspace pixel: (5); in: represents the updated hyperspace feature of each hyperspace pixel, Represents the weight matrix corresponding to the features before the self-attention information interaction in the hyperspace pixel.

[0013] Furthermore, in step S34, the features calculated based on the internal attention of the hyperspectral channel are reprojected according to formula (6) to obtain the updated hyperspectral features of each hyperspectral channel: (6); in: represents the updated hyperspectral features of each hyperspectral channel, Represents the weight matrix corresponding to the features before self-attention information interaction in the hyperspectral channel.

[0014] A spectral and spatial aggregation-guided remote sensing hyperspectral sharpening fidelity system, used to execute any of the spectral and spatial aggregation-guided remote sensing hyperspectral sharpening fidelity methods described above, comprising a hyperspatial clustering module, a hyperspectral clustering module, a hyperspatial interactive attention module, a hyperspectral interactive attention module, a hyperspatial self-attention module, a hyperspectral self-attention module, a hyperspatial feature aggregation module, a hyperspectral feature aggregation module and a remote sensing hyperspectral sharpening fidelity image fusion module; The hyperspace clustering module is used to divide the original space of the remote sensing hyperspectral image in the training set into multiple hyperspace pixel grids, and obtain the corresponding hyperspace pixel and space pixel affinity matrix through clustering iteration; The hyperspectral clustering module is used to divide the spectral channels of the remote sensing hyperspectral images in the training set into multiple hyperspectral channel grids, and obtain the corresponding hyperspectral channels and spectral affinity matrix through clustering iteration; The hyperspatial interactive attention module is used to interact the original spatial pixels of remote sensing hyperspectral images with the hyperspatial pixels and hyperspectral channels to obtain the interactive features of the hyperspatial pixels. The hyperspectral self-attention module is used to interact the original spectral channel of the remote sensing hyperspectral image with the hyperspectral channel and hyperspatial pixels to obtain the interactive features of the hyperspectral channel; The hyperspace feature aggregation module is used to aggregate the updated hyperspace features of all hyperspace pixels to obtain updated hyperspace features; The hyperspectral feature aggregation module is used to aggregate the updated hyperspectral features of all hyperspectral channels to obtain updated hyperspectral features; The remote sensing hyperspectral sharpening and fidelity image fusion module is used to fuse the updated hyperspatial features with the updated hyperspectral features to obtain the remote sensing hyperspectral sharpening and fidelity image.

[0015] Beneficial effects of the invention: The remote sensing hyperspectral sharpening and fidelity preservation method and system guided by spectral and spatial aggregation provided by the present invention have the following advantages: (1) By using the spectral clustering iteration and spatial clustering iteration methods, the characteristic information of similar regions within the spectrum and the characteristic information of similar regions within the spatial pixels can be effectively obtained, making full use of the information of spectral and spatial similar regions.

[0016] (2) Based on the clustering of similar spectra and spatial regions, an interactive attention mechanism is used to construct hyperspectral channels and hyperspatial pixels, which promotes the interaction of information in long-distance similar regions and enhances the efficiency of information utilization.

[0017] (3) Based on the self-attention mechanism, the most relevant feature information is selected to ensure efficient concentration of information interaction and eliminate interference and redundant calculations.

[0018] (4) Double-branch interactive guidance can more effectively utilize the spatial and spectral information of panchromatic and hyperspectral image data, and achieve better spectral and spatial fidelity effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the process of the present invention.

[0020] Figure 2 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0021] A remote sensing hyperspectral sharpening and fidelity preservation method guided by spectral and spatial aggregation includes the following steps, and its flow chart is as follows: Figure 1 As shown: S1: Divide the original space of the remote sensing hyperspectral image in the training set into multiple hyperspatial pixel grids, and obtain the corresponding hyperspatial pixels and spatial pixel affinity matrices through clustering iteration. Divide the spectral channels of the remote sensing hyperspectral image in the training set into multiple hyperspectral channel grids, and obtain the corresponding hyperspectral channels and spectral affinity matrices through clustering iteration. Specifically, the following method can be used to obtain the hyperspatial pixel and spatial pixel affinity matrix: S11: The original space of the remote sensing hyperspectral image in the training set is divided into regular grids, and the original single pixel value and the original space pixel affinity matrix are obtained by sampling; S12: After a set number of clustering iterations, a spatial pixel affinity matrix is ​​obtained, and all spatial pixels in the spatial pixel affinity matrix are weighted to obtain hyper-spatial pixels.

[0022] Assume that the original grid size is , then the number of pixels that can be calculated as hyperspace pixels is After multiple clustering iterations, the spatial affinity matrix can be obtained. ; in: Represents the height of the grid of the original spatial division of the remote sensing hyperspectral image, Represents the width of the grid of the original space division of the remote sensing hyperspectral image, represents the number of hyperspace pixels, Represents the height of remote sensing hyperspectral images, represents the width of the remote sensing hyperspectral image, Indicates Row, No. The spatial pixels of the columns, Indicates The pixels of the row, , represents a real vector space with dimension equal to the number of spectral channels, express After the iteration Columns of hyperspace pixels, represents the square of the two-norm, Represents a natural constant.

[0023] Hyperspatial pixels can be obtained by weighting spatial pixels, as shown in the following formula: ; in: express After the iteration Columns of hyperspace pixels, represents the normalization of spatial pixels along columns, .

[0024] Specifically, the following method can be used to obtain the hyperspectral channel and spectral affinity matrix: S13: dividing the spectral channels of the remote sensing hyperspectral images in the training set into grids regularly to obtain the original spectral affinity matrix; S14: After a set number of clustering iterations, a spectral affinity matrix is ​​obtained, and all spectral channels in the spectral affinity matrix are weighted to obtain a hyperspectral channel.

[0025] Assume that the grid size of the divided hyperspectral channel is , so the number of hyperspectral channels can be calculated as ,in represents the number of hyperspectral channels, Represents the total number of spectral channels of remote sensing hyperspectral images; After clustering iterations, the spectral affinity matrix and the final hyperspectral channel can also be obtained: ; ; in: represents the spectral affinity matrix, Indicates The number of spectral channels in a row, express After the iteration The hyperspectral channels of the columns, express After the iteration The hyperspectral channels of the columns, represents the normalization of spectral channels along the columns.

[0026] Different from the previous method of processing the input image in regular blocks, this application divides the image into individual superpixels, and obtains the spatial pixel affinity matrix and hyperspatial pixels through continuous clustering iterations, which can take into account the similarities between pixels and connect them together. Compared with the regular segmentation method, this can better reconstruct accurate boundaries and reduce the blur of boundaries. In addition, in the process of spatial aggregation, only the correlation between each spatial pixel and the surrounding superpixels is calculated, which ensures the locality of the hyperspatial pixels, so that it can have higher computational efficiency in terms of both computing and memory. Applying the idea of ​​superpixel clustering to spectral channels can produce the same effect on spectral channels.

[0027] S2: The original spatial pixels of the remote sensing hyperspectral image are interacted with the hyperspatial pixels and the hyperspectral channels to obtain the interactive features of the hyperspatial pixels; the original spectral channels of the remote sensing hyperspectral image are interacted with the hyperspectral channels and the hyperspatial pixels to obtain the interactive features of the hyperspectral channels; Furthermore, in step S2, the original spatial pixels of the remote sensing hyperspectral image are interacted with the hyperspatial pixels and the hyperspectral channels by the following method to obtain the interactive features of the hyperspatial pixels: S21: The query, key and value of the hyperspace pixel are calculated based on the linear projection. Then, the interactive attention mechanism is used to calculate the preliminary hyperspace pixel interaction features after the original space pixel and the hyperspace pixel interact with each other according to formula (1): (1); in: represents the preliminary hyperspatial pixel interaction features, represents the normalized exponential function, represents a query of a hyperspace pixel, represents the key of the hyperspace pixel, represents the value of the hyperspace pixel, represents the scaling factor, Represents matrix transpose; Here the query of hyperspace pixels , ,in represents a real vector space whose dimension is the spatial dimension multiplied by the number of spectral channels, represents the given hyperspatial pixel feature, The weight matrix corresponding to the query of the hyperspace pixel; the key of the hyperspace pixel , , Represents a real vector space whose dimensions are the height times the width times the number of spectral channels of a remote sensing hyperspectral image. represents the original spatial pixel features, The weight matrix corresponding to the key of the hyperspace pixel; the value of the hyperspace pixel , , Represents the weight matrix corresponding to the value of the hyperspace pixel; S22: Re-acquire the query of the hyperspace pixel from the original space pixel feature, and use the key of the hyperspectral channel as the key and the preliminary hyperspace pixel interaction feature as the value, and use the interactive attention mechanism to further interact the hyperspace pixel features after the preliminary information interaction with the original space, so as to obtain the interactive features of the hyperspace pixel after the original space pixel interacts with the hyperspace pixel and the hyperspectral channel.

[0028] This step is different from the hyperspace pixel clustering in step S1 in that this process is not constrained by the neighborhood, and the interaction characteristics of the hyperspace pixels obtained are the long-distance interaction information of the spatial pixels, thus realizing the interaction of the long-distance information of the spatial pixels.

[0029] Furthermore, in step S2, the original spectral channel of the remote sensing hyperspectral image is interacted with the hyperspectral channel and the hyperspatial pixel by the following method to obtain the interactive features of the hyperspectral channel: S23: The query of the hyperspectral channel, the key of the hyperspectral channel and the value of the spectral channel are calculated according to the linear projection, and then the interactive attention mechanism is used to calculate the preliminary hyperspectral channel interaction features after the original spectral channel and the hyperspectral channel interact with each other according to formula (2): (2); in: represents the preliminary hyperspectral channel interaction characteristics, represents the normalized exponential function, represents the query of hyperspectral channels, represents the key of the hyperspectral channel, represents the value of the hyperspectral channel, represents the scaling factor, Represents matrix transpose; Here is the query of hyperspectral channel , , Represents a real vector space whose dimensions are the height times the width times the spatial dimension of the remote sensing hyperspectral image. represents the number of hyperspectral channels, Represents the given hyperspectral channel characteristics, Represents the weight matrix corresponding to the query of the hyperspectral channel; the key of the hyperspectral channel , , Represents a real vector space whose dimensions are the height times the width times the number of spectral channels of a remote sensing hyperspectral image. represents the spectral channel, , The weight matrix corresponding to the key of the hyperspectral channel; the value of the hyperspectral channel , , Represents the weight matrix corresponding to the values ​​of the hyperspectral channel; S24: Re-acquire the query of the hyperspectral channel from the original spectral channel, and use the key of the hyperspace pixel as the key and the preliminary hyperspectral channel interaction feature as the value, and use the interactive attention mechanism to further interact the hyperspectral channel features after the preliminary information interaction with the original spectral channel to obtain the interactive features of the hyperspectral channel in which the original spectral channel interacts with the hyperspectral channel and the hyperspace pixel.

[0030] This step is different from the hyperspectral channel aggregation in step S1 in that this process is not constrained by the neighborhood, and the interaction characteristics of the obtained hyperspectral channels are the long-distance interaction information of the spectral channels, thus realizing the interaction of the long-distance information of the spectral channels.

[0031] Through the hyperspatial and hyperspectral clustering operations in step S1, we can capture the local correlation of spatial pixels and spectral channels in the region. However, for the hyperspectral panchromatic sharpening task, this may lack the ability to capture the remote dependencies of spatial pixels and spectral channels. Therefore, we use the idea of ​​attention mechanism to enhance the interaction of spatial pixels and spectral channels for remote information, which can greatly utilize the information correlation between features and generate higher quality images. Because there is a great correlation between spatial pixels or spectral channels and their hyperspatial pixels or hyperspectral channels, the use of interactive attention can make full use of pixel or spectral information as much as possible, ensuring the interaction of long-distance information and enhancing the efficiency of information utilization.

[0032] S3: Based on the interactive features of hyperspace pixels, multiple pixels that are most similar to each hyperspace pixel are selected according to the spatial pixel affinity matrix, and the updated hyperspace features of each hyperspace pixel are obtained according to the self-attention mechanism. Based on the interactive features of hyperspectral channels, multiple spectral channels that are most similar to each hyperspectral channel are selected according to the spectral affinity matrix, and the updated hyperspectral features of each hyperspectral channel are obtained according to the self-attention mechanism. Specifically, the following method can be used to obtain the updated hyperspace features of each hyperspace pixel: S31: Sort the pixels in the spatial pixel affinity matrix in descending order, and select the first According to the self-attention mechanism, each pixel is linearly transformed to obtain the query of the self-attention space pixel, the key of the self-attention space pixel, and the value of the self-attention space pixel. Then, according to formula (3), the attention calculation is performed on the internal part of the superpixel to obtain the feature after the attention calculation of the internal part of the superspace pixel: (3); in: represents the features after attention calculation inside the hyperspace pixel, represents the query of pixels in the self-attention space, , represents the weight matrix corresponding to the query of the self-attention space pixel, represents the key of the self-attention space pixel, , represents the weight matrix corresponding to the key of the self-attention space pixel, represents the value of the pixel in the self-attention space, , Represents the weight matrix corresponding to the value of the pixel in the self-attention space; S32: Re-projection calculation is performed based on the features calculated based on the internal attention of the hyperspace pixel to obtain the updated hyperspace features of each hyperspace pixel.

[0033] Specifically, the features calculated based on the internal attention of the hyperspace pixel can be reprojected according to formula (5) to obtain the updated hyperspace features of each hyperspace pixel: (5); in: represents the updated hyperspace feature of each hyperspace pixel, Represents the weight matrix corresponding to the features before the self-attention information interaction in the hyperspace pixel.

[0034] Further, in step S3, the updated hyperspectral features of each hyperspectral channel are obtained by the following method: S33: Sort the spectral channels in the spectral affinity matrix in descending order, and select the first According to the self-attention mechanism, the spectral channel is linearly transformed to obtain the query of the self-attention spectral channel, the key of the self-attention spectral channel, and the value of the self-attention spectral channel. Then, according to formula (4), the attention calculation is performed inside the spectral channel to obtain the features after the attention calculation inside the hyper-spectral channel: (4); in: represents the features after attention calculation inside the hyperspectral channel, represents the query of the self-attention spectral channel, , represents the weight matrix corresponding to the query of the internal attention spectrum channel, represents the key of the self-attention spectral channel, , The weight matrix corresponding to the key representing the internal attention spectral channel, represents the value of the self-attention spectral channel, , Represents the weight matrix corresponding to the values ​​of the internal attention spectrum channels; S34: Re-projection calculation is performed based on the features calculated after the internal attention of the hyperspectral channel to obtain the updated hyperspectral features of each hyperspectral channel.

[0035] Furthermore, in step S34, the features calculated based on the internal attention of the hyperspectral channel are reprojected according to formula (6) to obtain the updated hyperspectral features of each hyperspectral channel: (6); in: represents the updated hyperspectral features of each hyperspectral channel, Represents the weight matrix corresponding to the features before self-attention information interaction in the hyperspectral channel.

[0036] S4: Aggregate the updated hyperspace features of all hyperspace pixels to obtain updated hyperspace features, and aggregate the updated hyperspectral features of all hyperspectral channels to obtain updated hyperspectral features; Since different hyperspatial pixels or hyperspectral channels have different numbers of attached spatial pixels or spectral channels, some may have a large number of spatial pixels or spectral channels, which will affect the use of complementary information of similar pixels or channels in the hyperspatial pixels or hyperspectral channels, and will result in additional computation and storage consumption.

[0037] Therefore, the present invention adopts the above method to perform self-attention processing, which can select the most relevant feature information, thereby prompting the network to focus more on effective information, ensuring efficient concentration of information interaction, and eliminating interference and redundant calculations.

[0038] S5: Fuse the updated hyperspatial features with the updated hyperspectral features to obtain a remote sensing hyperspectral sharpened fidelity image.

[0039] By constructing a dual-branch interactively guided network, the present invention can more effectively utilize the spatial information and spectral information of panchromatic image and hyperspectral image data, thereby achieving better spectral and spatial fidelity effects.

[0040] In order to verify that the present invention has better effects, we conducted quantitative index comparison experiments on the public data set Pavia Center with multiple most advanced algorithms in the world. The experimental results are shown in Table 1: Table 1:

[0041] It can be seen from Table 1 that the quantitative indicators of the present invention are better than those of the international advanced algorithms in recent years.

[0042] A spectral and spatial aggregation guided remote sensing hyperspectral sharpening fidelity system is used to execute any of the above spectral and spatial aggregation guided remote sensing hyperspectral sharpening fidelity methods. The system structure diagram is as follows Figure 2 As shown, it includes a hyperspace clustering module, a hyperspectral clustering module, a hyperspace interactive attention module, a hyperspectral interactive attention module, a hyperspace self-attention module, a hyperspectral self-attention module, a hyperspace feature aggregation module, a hyperspectral feature aggregation module and a remote sensing hyperspectral sharpening and fidelity image fusion module; The hyperspace clustering module is used to divide the original space of the remote sensing hyperspectral image in the training set into multiple hyperspace pixel grids, and obtain the corresponding hyperspace pixel and space pixel affinity matrix through clustering iteration; The hyperspectral clustering module is used to divide the spectral channels of the remote sensing hyperspectral images in the training set into multiple hyperspectral channel grids, and obtain the corresponding hyperspectral channels and spectral affinity matrix through clustering iteration; The hyperspatial interactive attention module is used to interact the original spatial pixels of the remote sensing hyperspectral image with the hyperspatial pixels and hyperspectral channels to obtain the interactive features of the hyperspatial pixels. The hyperspectral self-attention module is used to interact the original spectral channel of the remote sensing hyperspectral image with the hyperspectral channel and hyperspatial pixels to obtain the interactive features of the hyperspectral channel; The hyperspace feature aggregation module is used to aggregate the updated hyperspace features of all hyperspace pixels to obtain updated hyperspace features; The hyperspectral feature aggregation module is used to aggregate the updated hyperspectral features of all hyperspectral channels to obtain updated hyperspectral features; The remote sensing hyperspectral sharpening and fidelity image fusion module is used to fuse the updated hyperspatial features with the updated hyperspectral features to obtain the remote sensing hyperspectral sharpening and fidelity image.

[0043] In summary, the remote sensing hyperspectral sharpening and fidelity preservation method and system guided by spectral and spatial aggregation provided by the present invention achieves the improvement of spatial resolution while ensuring the integrity of spectral dimension by designing spatial and spectral feature aggregation and interaction mechanism, constructs a fusion module with noise immunity characteristics, can effectively deal with the interference caused by differences in multi-source data, and provides strong technical support for the analysis and practical application of remote sensing hyperspectral images.

[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A remote sensing hyperspectral sharpening and fidelity preservation method guided by spectral and spatial aggregation, characterized by: The steps include: S1: Divide the original space of the remote sensing hyperspectral image in the training set into multiple hyperspatial pixel grids, and obtain the corresponding hyperspatial pixels and spatial pixel affinity matrices through clustering iteration. Divide the spectral channels of the remote sensing hyperspectral image in the training set into multiple hyperspectral channel grids, and obtain the corresponding hyperspectral channels and spectral affinity matrices through clustering iteration. S2: The original spatial pixels of the remote sensing hyperspectral image are interacted with the hyperspatial pixels and the hyperspectral channels to obtain the interactive features of the hyperspatial pixels; the original spectral channels of the remote sensing hyperspectral image are interacted with the hyperspectral channels and the hyperspatial pixels to obtain the interactive features of the hyperspectral channels; S3: Based on the interactive features of hyperspace pixels, multiple pixels that are most similar to each hyperspace pixel are selected according to the spatial pixel affinity matrix, and the updated hyperspace features of each hyperspace pixel are obtained according to the self-attention mechanism. Based on the interactive features of hyperspectral channels, multiple spectral channels that are most similar to each hyperspectral channel are selected according to the spectral affinity matrix, and the updated hyperspectral features of each hyperspectral channel are obtained according to the self-attention mechanism. S4: Aggregate the updated hyperspace features of all hyperspace pixels to obtain updated hyperspace features, and aggregate the updated hyperspectral features of all hyperspectral channels to obtain updated hyperspectral features; S5: Fuse the updated hyperspatial features with the updated hyperspectral features to obtain a remote sensing hyperspectral sharpened fidelity image.

2. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 1, characterized in that: In step S1, the following method is used to obtain the hyperspatial pixel and spatial pixel affinity matrix: S11: The original space of the remote sensing hyperspectral image in the training set is divided into regular grids, and the original single pixel value and the original space pixel affinity matrix are obtained by sampling; S12: After a set number of clustering iterations, a spatial pixel affinity matrix is ​​obtained, and all spatial pixels in the spatial pixel affinity matrix are weighted to obtain hyper-spatial pixels.

3. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 1, characterized in that: In step S1, the following method is used to obtain the hyperspectral channel and spectral affinity matrix: S13: dividing the spectral channels of the remote sensing hyperspectral images in the training set into grids regularly to obtain the original spectral affinity matrix; S14: After a set number of clustering iterations, a spectral affinity matrix is ​​obtained, and all spectral channels in the spectral affinity matrix are weighted to obtain a hyperspectral channel.

4. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 1, characterized in that: In step S2, the original spatial pixels of the remote sensing hyperspectral image are interacted with the hyperspatial pixels and the hyperspectral channels by the following method to obtain the interactive features of the hyperspatial pixels: S2 1: Based on the linear projection, the query, key and value of the hyperspace pixel are calculated. Then, the interactive attention mechanism is used to calculate the preliminary hyperspace pixel interaction features after the original space pixel and the hyperspace pixel interact with each other according to formula (1): （1）； in: represents the preliminary hyperspatial pixel interaction features, represents the normalized exponential function, represents a query of a hyperspace pixel, represents the key of the hyperspace pixel, represents the value of the hyperspace pixel, represents the scaling factor, Represents matrix transpose; S22: Re-acquire the query of the hyperspace pixel from the original space pixel feature, and use the key of the hyperspectral channel as the key and the preliminary hyperspace pixel interaction feature as the value, and use the interactive attention mechanism to further interact the hyperspace pixel features after the preliminary information interaction with the original space, so as to obtain the interactive features of the hyperspace pixel after the original space pixel interacts with the hyperspace pixel and the hyperspectral channel.

5. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 1, characterized in that: In step S2, the original spectral channel of the remote sensing hyperspectral image is interacted with the hyperspectral channel and the hyperspatial pixel by the following method to obtain the interactive features of the hyperspectral channel: S23: The query of the hyperspectral channel, the key of the hyperspectral channel and the value of the spectral channel are calculated according to the linear projection, and then the interactive attention mechanism is used to calculate the preliminary hyperspectral channel interaction features after the original spectral channel and the hyperspectral channel interact with each other according to formula (2): （2）； in: represents the preliminary hyperspectral channel interaction characteristics, represents the normalized exponential function, represents the query of hyperspectral channels, represents the key of the hyperspectral channel, represents the value of the hyperspectral channel, represents the scaling factor, Represents matrix transpose; S24: Re-acquire the query of the hyperspectral channel from the original spectral channel, and use the key of the hyperspace pixel as the key and the preliminary hyperspectral channel interaction feature as the value, and use the interactive attention mechanism to further interact the hyperspectral channel features after the preliminary information interaction with the original spectral channel to obtain the interactive features of the hyperspectral channel in which the original spectral channel interacts with the hyperspectral channel and the hyperspace pixel.

6. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 4, characterized in that: In step S3, the updated hyperspace feature of each hyperspace pixel is obtained by the following method: S31: Sort the pixels in the spatial pixel affinity matrix in descending order, and select the first According to the self-attention mechanism, each pixel is linearly transformed to obtain the query of the self-attention space pixel, the key of the self-attention space pixel, and the value of the self-attention space pixel. Then, according to formula (3), the attention calculation is performed on the internal part of the superpixel to obtain the feature after the attention calculation of the internal part of the superspace pixel: （3）； in: represents the features after attention calculation inside the hyperspace pixel, represents the query of pixels in the self-attention space, represents the key of the self-attention space pixel, Represents the value of the pixel in the self-attention space; S32: Re-projection calculation is performed based on the features calculated based on the internal attention of the hyperspace pixel to obtain the updated hyperspace features of each hyperspace pixel.

7. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 5, characterized in that: In step S3, the updated hyperspectral features of each hyperspectral channel are obtained by the following method: S33: Sort the spectral channels in the spectral affinity matrix in descending order, and select the first According to the self-attention mechanism, the spectral channel is linearly transformed to obtain the query of the self-attention spectral channel, the key of the self-attention spectral channel, and the value of the self-attention spectral channel. Then, according to formula (4), the attention calculation is performed inside the spectral channel to obtain the features after the attention calculation inside the hyper-spectral channel: （4）； in: represents the features after attention calculation inside the hyperspectral channel, represents the query of the self-attention spectral channel, represents the key of the self-attention spectral channel, represents the value of the self-attention spectral channel; S34: Re-projection calculation is performed based on the features calculated after the internal attention of the hyperspectral channel to obtain the updated hyperspectral features of each hyperspectral channel.

8. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 6, characterized in that: In step S32, the features calculated based on the internal attention of the hyperspace pixel are reprojected according to formula (5) to obtain the updated hyperspace features of each hyperspace pixel: （5）； in: represents the updated hyperspace feature of each hyperspace pixel, Represents the weight matrix corresponding to the features before the self-attention information interaction in the hyperspace pixel.

9. The method for remote sensing hyperspectral sharpening and fidelity preservation guided by spectral and spatial aggregation according to claim 7, characterized in that: In step S34, the features calculated based on the internal attention of the hyperspectral channel are reprojected according to formula (6) to obtain the updated hyperspectral features of each hyperspectral channel: （6）； in: represents the updated hyperspectral features of each hyperspectral channel, Represents the weight matrix corresponding to the features before self-attention information interaction in the hyperspectral channel.

10. A remote sensing hyperspectral sharpening and fidelity preservation system guided by spectral and spatial aggregation, used to execute a remote sensing hyperspectral sharpening and fidelity preservation method guided by spectral and spatial aggregation as claimed in any one of claims 1 to 9, characterized in that: Including hyperspatial clustering module, hyperspectral clustering module, hyperspatial interactive attention module, hyperspectral interactive attention module, hyperspatial self-attention module, hyperspectral self-attention module, hyperspatial feature aggregation module, hyperspectral feature aggregation module and remote sensing hyperspectral sharpening and fidelity image fusion module; The hyperspace clustering module is used to divide the original space of the remote sensing hyperspectral image in the training set into multiple hyperspace pixel grids, and obtain the corresponding hyperspace pixel and space pixel affinity matrix through clustering iteration; The hyperspectral clustering module is used to divide the spectral channels of the remote sensing hyperspectral images in the training set into a plurality of hyperspectral channel grids, and obtain the corresponding hyperspectral channels and spectral affinity matrix through clustering iteration; The hyperspace interactive attention module is used to interact information between the original spatial pixels of the remote sensing hyperspectral image and the hyperspace pixels and hyperspectral channels to obtain interactive features of the hyperspace pixels; The hyperspectral self-attention module is used to perform information interaction between the original spectral channel of the remote sensing hyperspectral image and the hyperspectral channel and hyperspatial pixels to obtain the interactive features of the hyperspectral channel; The hyperspace feature aggregation module is used to aggregate the updated hyperspace features of all hyperspace pixels to obtain updated hyperspace features; The hyperspectral feature aggregation module is used to aggregate the updated hyperspectral features of all hyperspectral channels to obtain updated hyperspectral features; The remote sensing hyperspectral sharpening and fidelity image fusion module is used to fuse the updated hyperspatial features with the updated hyperspectral features to obtain a remote sensing hyperspectral sharpening and fidelity image.

Images