Hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation

Abstract

The present invention discloses a method for fusing hyperspectral and multispectral images based on multiscale spatial-spectral transformation. The method comprises: using a dual-branch network structure to perform cross-branch fusion before fusion to enhance the correlation between features extracted by the dual-branch network; employing multi-head self-attention based on spatial domain partitioning to extract spectral features, effectively acquiring the global characteristics of the spectrum; employing multi-scale patches to extract spatial features to better exploit the rich spatial features in the multispectral image; and then fusing the spectral and spatial features and performing image reconstruction to obtain a high-resolution hyperspectral image. Compared with existing technologies, this method fully considers the correlation between the feature maps of the multispectral image and the hyperspectral image, extracts features with high accuracy, and can produce high-quality, high-resolution hyperspectral images.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation. Background Art

[0002] When capturing remote sensing images, current physical imaging systems cannot simultaneously achieve high spatial and spectral resolutions. They can only capture images with high spatial resolution and limited spectral bands (high-resolution multispectral images), or images with low spatial resolution but higher spectral resolution (low-resolution hyperspectral images). Therefore, in practice, it is necessary to fuse the high spatial resolution of multispectral images with the high spectral resolution of hyperspectral images to generate high-resolution hyperspectral images (HR-HSI).

[0003] Although deep neural networks are currently used to learn the spectral and spatial details of remote sensing images and achieve good fusion performance, the correlation between the feature maps of multispectral images and hyperspectral images is still not taken into account, and the complementary features that may exist between multispectral images and hyperspectral images are ignored. The interaction between the features of multispectral images and hyperspectral images is weak, and the feature extraction accuracy is not high, resulting in low image quality.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method, device, intelligent terminal and computer-readable storage medium for the fusion of hyperspectral and multispectral images based on multi-scale spatial-spectral transformation, aiming to solve the problems in the prior art of weak interaction between multispectral and hyperspectral image features and low feature extraction accuracy.

[0006] To achieve the above objectives, the present invention provides a method for fusion of hyperspectral and multispectral images based on multi-scale spatial-spectral transformation, comprising:

[0007] The hyperspectral image and the multispectral image are each input into a branch of the dual-branch network and fused across the branches to obtain a fused hyperspectral image and a fused multispectral image;

[0008] Performing feature extraction on the fused hyperspectral image and the fused multispectral image respectively to obtain spectral shallow features and spatial shallow features;

[0009] Inputting the shallow spectral features into a first Transformer module to obtain deep spectral features, wherein the first Transformer module is provided with multi-head self-attention based on spatial domain partitioning;

[0010] Inputting the spatial shallow features into a second Transformer module to obtain spatial deep features, the second Transformer module uses multi-scale patches to extract spatial features in the multispectral image at multiple scales;

[0011] fusing the spectral depth feature and the spatial depth feature to obtain a spatial spectral feature;

[0012] The spatial spectral features are input into an image reconstruction module to obtain a high-resolution hyperspectral image.

[0013] Optionally, the step of inputting the hyperspectral image and the multispectral image into one branch of a dual-branch network and fusing them across the branches to obtain a fused hyperspectral image and a fused multispectral image includes:

[0014] Upsampling the hyperspectral image to obtain a sampled hyperspectral image;

[0015] Downsampling the multispectral image to obtain a sampled multispectral image;

[0016] Connecting the multispectral image and the sampled hyperspectral image to obtain a fused multispectral image;

[0017] The hyperspectral image and the sampled multispectral image are connected to obtain a fused hyperspectral image.

[0018] Optionally, inputting the shallow spectral features into a first Transformer module to obtain deep spectral features includes:

[0019] Flattening the shallow features of the spectrum to obtain a two-dimensional matrix;

[0020] Performing linear projection on the two-dimensional matrix to obtain a spectral token;

[0021] Inputting the spectral token into the first Transformer module to obtain spectral intermediate features;

[0022] The spectral intermediate features are convolved to obtain the spectral depth features.

[0023] Optionally, the first Transformer module includes multiple Transformer layers, and each Transformer layer is provided with multi-head self-attention based on spatial domain division.

[0024] Optionally, the second Transformer module includes a plurality of parallel feature extraction submodules, and the inputting of the spatial shallow features into the second Transformer module to obtain the spatial deep features includes:

[0025] Dividing the spatial shallow features into patches of different scales to obtain embedded features of patches of different scales;

[0026] Perform linear projection on all patch embedding features to obtain several spatial tokens, input each of the spatial tokens into a feature extraction submodule, and obtain the spatial depth features output by each feature extraction submodule;

[0027] Aggregate the spatial depth features output by all feature extraction submodules to obtain the spatial depth features.

[0028] Optionally, aggregating the spatial depth features output by all feature extraction submodules to obtain the spatial depth features includes:

[0029] The spatial depth features output by all feature extraction submodules are aggregated using learnable weights to obtain the spatial depth features.

[0030] Optionally, fusing the spectral depth feature and the spatial depth feature to obtain a spatial spectral feature includes:

[0031] Aggregating the shallow spectral features and the deep spectral features using skip connections to obtain spectral features;

[0032] Aggregating the spatial shallow features and the spatial deep features using skip connections to obtain spatial features;

[0033] Upsampling the spectral feature so that the feature dimension of the spectral feature is the same as the feature dimension of the spatial feature;

[0034] The spatial feature and the upsampled spectral feature are connected to obtain the spatial spectral feature.

[0035] In order to achieve the above object, the present invention also provides a hyperspectral and multispectral image fusion device based on multi-scale spatial spectrum transformation, the device comprising:

[0036] A cross-branch fusion module is used to input the hyperspectral image and the multispectral image into one branch of the dual-branch network and fuse them across the branches to obtain a fused hyperspectral image and a fused multispectral image;

[0037] A shallow feature module is used to extract features from the fused hyperspectral image and the fused multispectral image respectively to obtain spectral shallow features and spatial shallow features;

[0038] A deep feature module is configured to input the shallow spectral features into a first Transformer module to obtain deep spectral features. The first Transformer module is provided with multi-head self-attention based on spatial domain partitioning. The shallow spatial features are input into a second Transformer module to obtain deep spatial features. The second Transformer module uses multi-scale patches to extract spatial features from the multispectral image at multiple scales.

[0039] A depth feature fusion module, configured to fuse the spectral depth feature and the spatial depth feature to obtain a spatial spectral feature;

[0040] The reconstruction module is used to input the spatial spectrum characteristics into the image reconstruction module to obtain a high-resolution hyperspectral image.

[0041] In order to achieve the above-mentioned objectives, the present invention also provides an intelligent terminal, which includes a memory, a processor, and a hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation stored in the memory and runnable on the processor. When the hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation is executed by the processor, any step of the hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation is implemented.

[0042] In order to achieve the above-mentioned objectives, the present invention also provides a computer-readable storage medium, on which a hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation is stored. When the hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation is executed by a processor, any step of the hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation is implemented.

[0043] As can be seen from the above, the present invention's hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation first strengthens the correlation between features extracted by the dual-branch network through cross-branch fusion. Then, multi-head self-attention based on spatial domain partitioning is used to extract spectral features, improving the ability to extract highly correlated spectral features. Multi-scale patches are used to extract spatial features at multiple scales, which can better explore the rich spatial features in the multispectral image. Finally, the spectral features and spatial features are fused and reconstructed to obtain a high-resolution hyperspectral image. Compared with the existing technology, this method fully considers the correlation between the multispectral image and the hyperspectral image feature map, extracts features with high accuracy, and can obtain high-quality, high-resolution hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is a flow chart of an embodiment of a method for fusion of hyperspectral and multispectral images based on multi-scale spatial-spectral transformation provided by the present invention;

[0046] Figure 2 yes Figure 1 A schematic diagram of a network model of an embodiment;

[0047] Figure 3 yes Figure 1 Detailed flowchart of step S300 of the embodiment;

[0048] Figure 4 yes Figure 1 Detailed flowchart of step S400 of the embodiment;

[0049] Figure 5 Schematic diagram of the structure of a hyperspectral and multispectral image fusion device based on multi-scale spatial-spectral transformation provided by an embodiment of the present invention;

[0050] Figure 6 This is a block diagram of the internal structure of the smart terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0052] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0053] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0054] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0055] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0056] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0058] Hyperspectral remote sensing images are usually taken from high altitudes and space by aircraft or satellites. Therefore, a hyperspectral remote sensing image often covers a very large area, which leads to the generally small spatial resolution of hyperspectral remote sensing images, and the area represented by one pixel is also very large. However, due to the limitations of current physical imaging systems, there is an inevitable trade-off, and high spatial resolution and high spectral resolution cannot be obtained at the same time. The imaging mechanism can only capture images with high spatial resolution and limited spectral bands, such as high-resolution multispectral images; or images with low spatial resolution but higher spectral resolution, such as low-resolution hyperspectral images. Therefore, in practice, it is necessary to fuse the high spatial resolution of multispectral images and the high spectral resolution of hyperspectral images to generate high-resolution hyperspectral images (HR-HSI).

[0059] Although existing technologies have used deep neural networks to learn the spectral and spatial details of remote sensing images and achieved good fusion performance, such as constructing unsupervised sparse Dirichlet networks and using unsupervised encoder-decoder architectures to extract and fuse spatial and spectral information from two modalities of different dimensions, they still do not consider the correlation between the feature maps of multispectral images and hyperspectral images, and ignore the complementary features that may exist between multispectral images and hyperspectral images. The interaction between the features of multispectral and hyperspectral images is weak, and the feature extraction accuracy is low, resulting in low image quality.

[0060] To address the aforementioned technical issues, the present invention utilizes a dual-branch network architecture with cross-branch fusion prior to fusion to enhance the correlation between features extracted by the dual-branch network. When extracting spectral features, multi-head self-attention based on spatial domain partitioning is employed to effectively capture the global characteristics of the spectrum. When extracting spatial features, multi-scale patching is employed to better exploit the rich spatial features in the multispectral image. Consequently, the present invention fully considers the correlation between the feature maps of multispectral and hyperspectral images, extracts features with high accuracy, and produces high-quality, high-resolution hyperspectral images.

[0061] Exemplary Methods

[0062] like Figure 1 As shown, an embodiment of the present invention provides a hyperspectral and multispectral image fusion method based on multi-scale spatial spectrum transformation, which is deployed on an intelligent terminal. Specifically, the method includes the following steps:

[0063] Step S100: inputting the hyperspectral image and the multispectral image into one branch of the dual-branch network respectively and fusing them across the branches to obtain a fused hyperspectral image and a fused multispectral image;

[0064] Specifically, it is necessary to extract spectral features and spatial features from hyperspectral images and multispectral images respectively. Therefore, the network model of this embodiment is a dual-branch network model. Figure 2 As shown in area (a), the system consists of two parallel, dual-branch networks: the upper branch processes hyperspectral images, and the lower branch processes multispectral images. The shallow spectral and spatial feature extractors are constructed using convolutional networks, while the deep spectral and spatial feature extractors are constructed using Transformer networks. The convolutional network extracts local features of the remote sensing image, followed by the Transformer network extracts global features, resulting in more accurate features.

[0065] Since the features extracted before fusion using the dual-branch network structure have the problem of weak correlation, the dual-branch network structure of this embodiment includes a cross-branch fusion module, which fuses the hyperspectral image and the multispectral image before inputting the feature extractor to enhance the correlation between the spatial features and spectral features (hereinafter referred to as: spatial-spectral features) extracted by the dual-branch network.

[0066] In this embodiment, cross-branch fusion specifically includes the following steps:

[0067] Step S110: upsampling the hyperspectral image to obtain a sampled hyperspectral image;

[0068] Step S120: downsampling the multispectral image to obtain a sampled multispectral image;

[0069] Step S130: connecting the multispectral image and the sampled hyperspectral image to obtain a fused multispectral image;

[0070] Step S140: connecting the hyperspectral image and the sampled multispectral image to obtain a fused hyperspectral image.

[0071] refer to Figure 2 ,assumed and Represent low-resolution hyperspectral images and high-resolution multispectral images respectively. First, bilinear interpolation is used to upsample the low-resolution hyperspectral image to obtain Then use bilinear interpolation to downsample the high-resolution multispectral image to obtain Then, in the spectral dimension, and and Connect to obtain fused hyperspectral image and fused multispectral images

[0072] It should be noted that the upsampling and downsampling methods are not limited, and nearest neighbor or bicubic interpolation may also be used.

[0073] Through cross-branch fusion connection, cross-modal information interaction between the two branches is achieved, which can enhance the correlation between the spatial spectral features extracted by the two-branch network and improve the quality of subsequent fusion images.

[0074] Step S200: performing feature extraction on the fused hyperspectral image and the fused multispectral image respectively to obtain spectral shallow features and spatial shallow features;

[0075] Specifically, the fused hyperspectral image and the fused multispectral image are each input into one branch of the dual-branch network for feature extraction to obtain spectral shallow features and spatial shallow features.

[0076] This embodiment uses a shallow spectral feature extractor and a shallow spatial feature extractor to extract shallow spectral features and shallow spatial features respectively. Since convolution is a simple and effective method to map an image to a higher-dimensional feature space, both the shallow spectral feature extractor and the shallow spatial feature extractor are constructed by convolutional networks. The size of the feature dimensions of the shallow spectral features and the shallow spatial features are uniformly mapped to C. The obtained shallow spectral features are The shallow spatial features are The size and number of convolution kernels are not limited. In this embodiment, the convolution kernel size is 3.

[0077] Step S300: Inputting shallow spectral features into a first Transformer module to obtain deep spectral features, wherein the first Transformer module is provided with multi-head self-attention based on spatial domain partitioning;

[0078] Specifically, to address the shortcoming of CNN-based unsupervised training networks, which can only obtain local features, this paper uses a Transformer network to better extract global spectral features from hyperspectral images. In the Transformer network, global self-attention is calculated in the spectral domain, and multiple heads are divided in the spatial domain to effectively obtain global spectral features.

[0079] In this embodiment, the first Transformer module is a deep spectral feature extractor, which is set on the branch for processing hyperspectral images. After that, it is input into the deep spectral feature extractor, which uses the Transformer's multi-head self-attention to better extract the global spectral features in the hyperspectral spectrum. Preferably, the multi-head self-attention mechanism of the deep spectral feature extractor is designed to be multi-head self-attention based on spatial domain partitioning, that is, each head is obtained by splitting according to the spatial dimension, which can more effectively obtain the global features of the spectrum.

[0080] Specifically, Transformer can be any Transformer used in the field of computer vision, such as the commonly used Swin Transformer, Vision Transformer, etc.

[0081] Step S400: Inputting the spatial shallow features into a second Transformer module to obtain spatial deep features. The second Transformer module uses multi-scale patches to extract spatial features in the multispectral image at multiple scales.

[0082] Specifically, the second Transformer module is set on the branch for processing multispectral images. After that, the image is fed into the second Transformer module to mine the rich spatial features in the multispectral image. Preferably, multi-scale patch embedding is used in the second Transformer module to extract multi-scale features. The final spatial features are then obtained by fusing the multi-scale features using learnable weights. The weights adaptively adjust the importance of each extracted feature, enabling multi-scale extraction of spatial features in the multispectral image.

[0083] Specifically, Transformer can be any Transformer used in the field of computer vision, such as the commonly used Swin Transformer, Vision Transformer, etc.

[0084] Step S500: fusing spectral depth features and spatial depth features to obtain spatial spectral features;

[0085] Specifically, long skip connections are used to aggregate spectral shallow features. and deep spectral features Get spectral characteristics Aggregate spatially shallow features using long skip connections and spatial depth features Get spatial features In order to ensure that the feature dimensions before reconstructing the image are the same, the spectral features Upsampling is performed to obtain Then connect and Get spatial spectral characteristics

[0086] There is no limit to the method for upsampling the spectral features, and various upsampling methods such as sub-pixel convolution layer, transposed convolution, and predefined interpolation can be used.

[0087] Step S600: inputting the spatial spectrum characteristics into the image reconstruction module to obtain a high-resolution hyperspectral image.

[0088] Specifically, the obtained spatial spectral features are finally input into the image reconstruction module to obtain a high-resolution hyperspectral image For example, a convolutional neural network is used to deconvolve features to reconstruct an image. Model parameters are optimized using reconstruction and adversarial losses. Training samples are constructed from a large number of images, and the designed deep convolutional neural network is trained to achieve image reconstruction. Feature-based image reconstruction methods are conventional techniques in the field and will not be discussed further here.

[0089] As described above, by using a dual-branch network structure, cross-branch fusion is performed before fusion to strengthen the correlation between the features extracted by the dual-branch network; multi-head self-attention based on spatial domain partitioning is used to effectively obtain global spectral features; multi-scale patches are used to better explore the rich spatial features in the multispectral image, and spectral and spatial features are then fused and reconstructed to obtain a high-resolution hyperspectral image. Therefore, by fully considering the correlation between the feature maps of the multispectral image and the hyperspectral image, the feature extraction accuracy is high, and a high-quality, high-resolution hyperspectral image can be obtained.

[0090] In one embodiment, Figure 3 As shown, the above step S300 specifically includes the following steps:

[0091] Step S310: Flattening the shallow features of the spectrum to obtain a two-dimensional matrix;

[0092] Step S320: Perform linear projection on the two-dimensional matrix to obtain spectral tokens;

[0093] Step S330: Input the spectral token into the first Transformer module to obtain the spectral intermediate features;

[0094] Step S340: Convolve the intermediate spectral features to obtain spectral depth features.

[0095] Specifically, before feeding the data into the deep feature extractor, the spectral shallow features are extracted. Flatten it into a two-dimensional matrix, and then perform linear projection on the two-dimensional matrix through a trainable linear projection to obtain the spectral token Then, Spectral Transformer (SpeT) is used to extract spectral intermediate features. Finally, the spectral depth feature is output using a convolution with a convolution kernel size of 3. refer to Figure 2 As shown in the figure, in this embodiment, the first Transformer module is a deep spectral feature extractor, which is constructed by the above-mentioned spectral Transformer. Specifically, it includes multiple SpeT layers, each of which is equipped with multi-head self-attention based on spatial domain partitioning.

[0096] The steps to obtain multi-head self-attention based on spatial domain division are: first input Get the query through trainable linear projection key and value Then, we calculate the similarity between the query and the key to get the attention weight, and apply the weight to the value to get the attention. The specific expression is:

[0097]

[0098]

[0099] Where W Q 、W K and W V is a learnable projection matrix, and Attention(·) is a scaled dot product operation. Split Q, K, and V into M along the spatial dimension 2 heads, the height and width of each head are h d =h / M and w d =w / M. The calculation formula of spectral multi-head self-attention (SpeMSA) is:

[0100] head m =Attention(Q m ,K m ,V m ),m=1…M 2

[0101]

[0102] in represents the m-th head, and W is a projection matrix with learnable parameters.

[0103] As described above, this embodiment proposes a Transformer that performs calculations in the spectral dimension. By using spectral multi-head self-attention, global self-attention is calculated in the spectral domain, and multi-head self-attention is divided in the spatial domain. This can effectively extract global spectral features from the hyperspectrum, thereby improving the network's ability to extract highly correlated spectral features in the hyperspectrum.

[0104] Since the existing fusion methods all use a single scale to extract spatial features. When using Transformer to extract global spatial features from multi-spectral images with rich spatial information, the fineness of the features obtained under patch embedding at different scales may be very different. Therefore, in one embodiment, a feature extraction method using multi-scale patch embedding is provided to obtain spatial features of different scales of multi-spectral images and combine them through adaptive weighting for subsequent fusion with spectral features. Figure 4 As shown, the specific steps include:

[0105] Step S410: Dividing spatial shallow features by patches of different scales to obtain embedded features of patches of different scales;

[0106] Step S420: linearly project all patch embedding features to obtain several spatial tokens, input each spatial token into a feature extraction submodule, and obtain the spatial depth features output by each feature extraction submodule;

[0107] Specifically, in this embodiment, the second Transformer module includes multiple feature extraction submodules, each of which is a deep spatial feature extractor. Each deep spatial feature extractor includes multiple spatial transformers (Spatial Transformer: SpaT). The shallow spatial features are divided into patches of different scales to obtain multiple patch embedding features of different scales. Linear projection is performed on each patch embedding feature to obtain a spatial token. where (p,p) is the size of the patch, is the number of tokens. Each spatial token is input into a deep spatial feature extractor, that is, the input of each deep spatial feature extractor is a patch embedding of different scales, and each layer of spatial Transformer outputs the spatial intermediate features in turn. Finally, a convolution kernel size of 3 is used for convolution processing to obtain the spatial depth features output by the deep spatial feature extractor.

[0108] Step S430: Aggregate the spatial depth features output by all feature extraction submodules to obtain the final spatial depth features.

[0109] Specifically, we use the learnable weights w l Aggregate the spatial depth features output by all deep spatial feature extractors Get the final spatial depth feature That is, the spatial depth feature output by the second Transformer module. The specific expression is:

[0110]

[0111] As described above, to address the issue of single-scale extracted features, this embodiment designs a multi-scale feature extraction branch. This branch first uses multi-scale patch embedding to extract multi-scale features, and then fuses these multi-scale features using learnable weights to obtain the final spatial depth features. These weights adaptively adjust the importance of each extracted feature, better mining the rich spatial features in the multi-spectral image.

[0112] In summary, this embodiment first uses cross-branch connections to strengthen the correlation between the spatial spectrum features extracted by the dual-branch network, and then extracts the shallow features of the spatial spectrum through two shallow feature extractors. Next, two deep feature extractors are used to extract the deep features of the spatial spectrum respectively. The two deep feature extractors are a deep spectral feature extractor and a deep spatial feature extractor. Among them, the deep spectral feature extractor contains multiple spectral transformers; the deep spatial feature extractor contains multiple spatial transformers, and finally the fused image is reconstructed through an image reconstruction module. Taking full account of the correlation between the feature maps of multispectral images and hyperspectral images, the feature extraction accuracy is high, and high-quality, high-resolution hyperspectral images can be obtained.

[0113] Exemplary devices

[0114] like Figure 5 As shown, corresponding to the hyperspectral and multispectral image fusion method based on multi-scale spatial spectrum transformation, the embodiment of the present invention also provides a hyperspectral and multispectral image fusion device based on multi-scale spatial spectrum transformation. Specifically, the device includes:

[0115] A cross-branch fusion module 600 is used to input the hyperspectral image and the multispectral image into one branch of the dual-branch network and fuse them across the branches to obtain a fused hyperspectral image and a fused multispectral image;

[0116] A shallow feature module 610 is used to extract features from the fused hyperspectral image and the fused multispectral image to obtain spectral shallow features and spatial shallow features;

[0117] A deep feature module 620 is configured to input the shallow spectral features into a first Transformer module to obtain deep spectral features. The first Transformer module is provided with multi-head self-attention based on spatial domain partitioning. The shallow spatial features are input into a second Transformer module to obtain deep spatial features. The second Transformer module uses multi-scale patches to extract spatial features from the multispectral image at multiple scales.

[0118] A depth feature fusion module 630 is configured to fuse the spectral depth feature and the spatial depth feature to obtain a spatial spectral feature;

[0119] The reconstruction module 640 is configured to input the spatial spectrum characteristics into an image reconstruction module to obtain a high-resolution hyperspectral image.

[0120] In this embodiment, the hyperspectral and multispectral image fusion device based on multi-scale spatial-spectral transformation can refer to the corresponding description of the hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation, which will not be repeated here.

[0121] Based on the above embodiment, the present invention also provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 6 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation. The internal memory provides an environment for the operation of the operating system and the hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation is executed by the processor, the steps of any one of the above-mentioned hyperspectral and multispectral image fusion methods based on multi-scale spatial-spectral transformation are implemented. The display screen of the intelligent terminal can be a liquid crystal display or an electronic ink display.

[0122] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention and does not constitute a limitation on the smart terminal to which the solution of the present invention is applied. The specific smart terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0123] In one embodiment, a smart terminal is provided. The smart terminal includes a memory, a processor, and a hyperspectral and multispectral image fusion program based on multiscale spatial-spectral transformation stored in the memory and executable on the processor. When the hyperspectral and multispectral image fusion program based on multiscale spatial-spectral transformation is executed by the processor, the following operating instructions are performed:

[0124] The hyperspectral image and the multispectral image are each input into a branch of the dual-branch network and fused across the branches to obtain a fused hyperspectral image and a fused multispectral image;

[0125] Performing feature extraction on the fused hyperspectral image and the fused multispectral image respectively to obtain spectral shallow features and spatial shallow features;

[0126] Inputting the shallow spectral features into a first Transformer module to obtain deep spectral features, wherein the first Transformer module is provided with multi-head self-attention based on spatial domain partitioning;

[0127] Inputting the spatial shallow features into a second Transformer module to obtain spatial deep features, the second Transformer module uses multi-scale patches to extract spatial features in the multispectral image at multiple scales;

[0128] fusing the spectral depth feature and the spatial depth feature to obtain a spatial spectral feature;

[0129] The spatial spectral features are input into an image reconstruction module to obtain a high-resolution hyperspectral image.

[0130] Optionally, the step of inputting the hyperspectral image and the multispectral image into one branch of a dual-branch network respectively and fusing them across the branches to obtain a fused hyperspectral image and a fused multispectral image includes: upsampling the hyperspectral image to obtain a sampled hyperspectral image; downsampling the multispectral image to obtain a sampled multispectral image; connecting the multispectral image and the sampled hyperspectral image to obtain a fused multispectral image; and connecting the hyperspectral image and the sampled multispectral image to obtain a fused hyperspectral image.

[0131] Optionally, inputting the shallow spectral features into the first Transformer module to obtain deep spectral features includes: flattening the shallow spectral features to obtain a two-dimensional matrix; performing linear projection on the two-dimensional matrix to obtain spectral tokens; inputting the spectral tokens into the first Transformer module to obtain intermediate spectral features; and convolving the intermediate spectral features to obtain the deep spectral features.

[0132] Optionally, the first Transformer module includes multiple Transformer layers, and each Transformer layer is provided with multi-head self-attention based on spatial domain division.

[0133] Optionally, the second Transformer module includes several parallel feature extraction sub-modules, and the spatial shallow features are input into the second Transformer module to obtain spatial depth features, including: dividing the spatial shallow features into patches of different scales to obtain patch embedding features of different scales; performing linear projection on all patch embedding features to obtain several spatial tokens, and inputting each of the spatial tokens into a feature extraction sub-module to obtain spatial depth features output by each feature extraction sub-module; and aggregating the spatial depth features output by all feature extraction sub-modules to obtain the spatial depth features.

[0134] Optionally, the aggregating the spatial depth features output by all feature extraction submodules to obtain the spatial depth features includes: using a learnable weight w l Aggregate the spatial depth features output by all feature extraction submodules to obtain the spatial depth features.

[0135] Optionally, the fusing of the spectral depth features and the spatial depth features to obtain spatial spectral features includes: using jump connections to aggregating the spectral shallow features and the spectral depth features to obtain spectral features; using jump connections to aggregating the spatial shallow features and the spatial depth features to obtain spatial features; upsampling the spectral features so that the feature dimensions of the spectral features are the same as the feature dimensions of the spatial features; and connecting the spatial features and the upsampled spectral features to obtain the spatial spectral features.

[0136] An embodiment of the present invention also provides a computer-readable storage medium, which stores a hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation. When the hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation is executed by a processor, the steps of any one of the hyperspectral and multispectral image fusion methods based on multi-scale spatial-spectral transformation provided in the embodiment of the present invention are implemented.

[0137] It should be understood that the sequence numbers of the steps in the above embodiments do not imply a specific order of execution; the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0138] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0139] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0140] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0141] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal device and method can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For example, the division of the modules or units described above is merely a logical functional division. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented.

[0142] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0143] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. Hyperspectral and multispectral image fusion method based on multi-scale spatial spectrum transformation, characterized by: include: The hyperspectral image and the multispectral image are each input into a branch of the dual-branch network and fused across the branches to obtain a fused hyperspectral image and a fused multispectral image; Performing feature extraction on the fused hyperspectral image and the fused multispectral image respectively to obtain spectral shallow features and spatial shallow features; Inputting the shallow spectral features into a first Transformer module to obtain deep spectral features, wherein the first Transformer module is provided with multi-head self-attention based on spatial domain partitioning; Inputting the spatial shallow features into a second Transformer module to obtain spatial deep features, the second Transformer module uses multi-scale patches to extract spatial features in the multispectral image at multiple scales; fusing the spectral depth feature and the spatial depth feature to obtain a spatial spectral feature; The spatial spectral features are input into an image reconstruction module to obtain a high-resolution hyperspectral image.

2. The hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation according to claim 1 is characterized in that: The step of inputting the hyperspectral image and the multispectral image into one branch of the dual-branch network and fusing them across the branches to obtain a fused hyperspectral image and a fused multispectral image comprises: Upsampling the hyperspectral image to obtain a sampled hyperspectral image; Downsampling the multispectral image to obtain a sampled multispectral image; Connecting the multispectral image and the sampled hyperspectral image to obtain a fused multispectral image; The hyperspectral image and the sampled multispectral image are connected to obtain a fused hyperspectral image.

3. The hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation according to claim 1 is characterized in that: Inputting the shallow spectral features into the first Transformer module to obtain deep spectral features includes: Flattening the shallow features of the spectrum to obtain a two-dimensional matrix; Performing linear projection on the two-dimensional matrix to obtain a spectral token; Inputting the spectral token into the first Transformer module to obtain spectral intermediate features; The spectral intermediate features are convolved to obtain the spectral depth features.

4. The hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation according to claim 3 is characterized in that: The first Transformer module includes multiple Transformer layers, each of which is equipped with multi-head self-attention based on spatial domain division.

5. The hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation according to claim 1 is characterized in that: The second Transformer module includes several parallel feature extraction submodules, and the inputting of the spatial shallow features into the second Transformer module to obtain spatial deep features includes: Dividing the spatial shallow features into patches of different scales to obtain embedded features of patches of different scales; Perform linear projection on all patch embedding features to obtain several spatial tokens, input each of the spatial tokens into a feature extraction submodule, and obtain the spatial depth features output by each feature extraction submodule; Aggregate the spatial depth features output by all feature extraction submodules to obtain the spatial depth features.

6. The hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation according to claim 5 is characterized in that: The aggregating the spatial depth features output by all feature extraction submodules to obtain the spatial depth features includes: The spatial depth features output by all feature extraction submodules are aggregated using learnable weights to obtain the spatial depth features.

7. The hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation according to claim 1, characterized in that: The fusing the spectral depth feature and the spatial depth feature to obtain a spatial spectral feature includes: Aggregating the shallow spectral features and the deep spectral features using skip connections to obtain spectral features; Aggregating the spatial shallow features and the spatial deep features using skip connections to obtain spatial features; Upsampling the spectral feature so that the feature dimension of the spectral feature is the same as the feature dimension of the spatial feature; The spatial feature and the upsampled spectral feature are connected to obtain the spatial spectral feature.

8. Hyperspectral and multispectral image fusion device based on multi-scale spatial spectrum transformation, characterized in that: The device comprises: A cross-branch fusion module is used to input the hyperspectral image and the multispectral image into one branch of the dual-branch network and fuse them across the branches to obtain a fused hyperspectral image and a fused multispectral image; A shallow feature module is used to extract features from the fused hyperspectral image and the fused multispectral image respectively to obtain spectral shallow features and spatial shallow features; A deep feature module is configured to input the shallow spectral features into a first Transformer module to obtain deep spectral features. The first Transformer module is provided with multi-head self-attention based on spatial domain partitioning. The shallow spatial features are input into a second Transformer module to obtain deep spatial features. The second Transformer module uses multi-scale patches to extract spatial features from the multispectral image at multiple scales. A depth feature fusion module, configured to fuse the spectral depth feature and the spatial depth feature to obtain a spatial spectral feature; The reconstruction module is used to input the spatial spectrum characteristics into the image reconstruction module to obtain a high-resolution hyperspectral image.

9. Intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation stored in the memory and runnable on the processor. When the hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation is executed by the processor, the steps of the hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation. When the hyperspectral and multispectral image fusion program based on multi-scale spatial-spectral transformation is executed by the processor, the steps of the hyperspectral and multispectral image fusion method based on multi-scale spatial-spectral transformation as described in any one of claims 1 to 7 are implemented.

Images