Multispectral and hyperspectral image fusion method based on depth separable convolution
By combining Swin Transformer and a two-branch two-level model with depth separation convolution, the adaptive fusion of multi-spectral and hyperspectral images is solved, and the problem of insufficient image fusion quality in the existing technology is achieved. High-quality and high-resolution image fusion effect.
Patent Information
- Application Number
- CN202411101248.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art is difficult to effectively fuse multispectral and hyperspectral remote sensing images, especially while maintaining spectral information integrity and improving spatial details, there is a lack of high-quality fusion methods for satellite images.
Using multi-spectral and hyperspectral image fusion method based on depth separable convolution, combined with Swin Transformer network and depth separable convolution layer, the global context and Adaptive fusion of local features.
It realizes high-quality multi-spectral and hyperspectral images, has strong spectral retention capabilities, avoids board effects and details loss, and improves spatial resolution and image quality.
Smart Images

Figure CN119068330A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of remote sensing image fusion, and in particular to a multispectral and hyperspectral image fusion method based on deep separable convolution. Background Art
[0002] Remote sensing image fusion is an economical and effective means of image quality enhancement. It can combine complementary information in different types of images to obtain more comprehensive and detailed observation data. Panchromatic and multispectral image fusion, also known as panchromatic sharpening, has always been a research hotspot in the field of remote sensing image fusion. With the development of artificial intelligence, in addition to the traditional commonly used component replacement and multi-resolution analysis, many new algorithms have emerged, such as those based on degradation models and deep learning. The former restores the fused image by optimizing the objective function, and the latter performs image fusion through sample learning methods.
[0003] In recent years, deep learning, in particular, has been widely used in the field of remote sensing image fusion due to its powerful feature extraction and feature expression capabilities. With the continuous development of satellite sensors, the emergence of hyperspectral remote sensing has broadened the data sources and fusion data types of remote sensing image fusion. Hyperspectral sensors have been widely used in the fields of fine classification of objects and environmental parameter inversion because they can obtain continuous spectral information of objects and have rich spatial detail features. Also due to the limitations of imaging spectral technology, the spatial resolution of tens of meters limits the further application of hyperspectral images. Hyperspectral remote sensing images usually have low spatial resolution, blurred spatial details and mixed pixels, which limits the application scope of images. Multispectral images and hyperspectral images of the same scene acquired by visible / near-infrared cameras and hyperspectral cameras on satellites can facilitate the fusion of the two. The fusion method of multispectral and hyperspectral images is similar to panchromatic sharpening, but how to design a better fusion method based on sensor characteristics and data characteristics to be fused to obtain fused images with high spatial and high spectral resolution is an urgent problem to be solved.
[0004] In remote sensing fusion images, reducing the spectral distortion generated during the fusion process, while improving the spatial details and maintaining the integrity of the spectral information are research hotspots in pixel-level remote sensing image fusion. Currently, the fusion algorithms for high-resolution multispectral images and low-resolution hyperspectral images are relatively mature, but most of the algorithms are based on aerial images or indoor data sets, which makes it difficult to apply them to satellite images. Existing deep learning methods have strong spectral preservation capabilities, but weak ability to enhance the spatial details inside and at the edges of images.
[0005] In summary, there is currently a lack of a method for multispectral and hyperspectral image fusion. Summary of the invention
[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a multispectral and hyperspectral image fusion method based on deep separable convolution to achieve high-quality fusion of multispectral and hyperspectral images.
[0007] The purpose of the present invention can be achieved by the following technical solutions:
[0008] In one aspect of the present invention, a multispectral and hyperspectral image fusion method based on deep separable convolution is provided, comprising the following steps:
[0009] Step S1, obtaining a hyperspectral remote sensing image and a multispectral remote sensing image to be fused and performing feature mapping respectively, and obtaining an input feature map by splicing;
[0010] Step S2, using the input feature map as input to a SwinTransformer network branch for extracting global context features and a separable convolutional layer branch for extracting local features, respectively, to obtain the original feature maps output by the two branches and the corresponding downsampled feature maps;
[0011] Step S3, concatenating the downsampled feature maps corresponding to the two branches and performing transposed convolution processing to obtain an upsampled feature map, combining the upsampled feature map and the original feature map, and using the spatial spectral attention mechanism to achieve adaptive feature fusion to obtain a fused feature map;
[0012] Step S4, obtaining a fused image based on the fused feature map.
[0013] As a preferred technical solution, in step S2, the process of inputting the input feature map into the SwinTransformer network branch to obtain the corresponding original feature map includes the following steps:
[0014] Step S201, performing block cutting and linear projection processing on the input feature map;
[0015] Step S202, inputting the feature map processed in step S201 into the Swin Transformer encoder based on window multi-head self-attention and moving window multi-head sub-attention to obtain the original feature map of the Swin Transformer network branch;
[0016] In step S203, the original feature map of the Swin Transformer network branch is inversely transformed and the image blocks are merged to achieve downsampling, and then input again into the Swin Transformer encoder based on window multi-head self-attention and moving window multi-head sub-attention to obtain the downsampled feature map of the Swin Transformer network branch.
[0017] As a preferred technical solution, in step S3, the process of combining the upsampled feature map and the original feature map to realize adaptive feature fusion using the spatial spectrum attention mechanism to obtain the fused features includes the following steps:
[0018] Step S301, concatenate the original feature maps output by the two branches and the upsampled feature map to obtain a first feature map, i.e. Figure 6 The leftmost feature map in (c);
[0019] Step S302, based on the first feature map, using channel attention calculation to obtain a channel attention map;
[0020] Step S303, concatenate the channel attention map and the first feature map to obtain a second feature map, that is, Figure 6 The first one from left to right in (c) The feature map at the location, based on the second feature map, using the spatial attention mechanism to calculate a spatial attention map;
[0021] Step S304: merge the second feature map and the spatial attention map to obtain a fused feature map.
[0022] As a preferred technical solution, in step S302, the process of obtaining a channel attention map by using channel attention calculation includes:
[0023] The feature map is processed by maximum pooling and average pooling along the spatial dimension, input into the fully connected layer, and added and calculated by the activation function to obtain the channel attention map.
[0024] As a preferred technical solution, in step S303, the process of calculating the spatial attention map using the spatial attention mechanism includes:
[0025] The two images are obtained by taking the maximum and average operations along the channel dimension. After the convolution layer operation, the output is a feature map with a channel number of 1, and after the activation function, the spatial attention map is obtained.
[0026] As a preferred technical solution, in step S4, a 1×1 convolution is used to adjust the number of output channels of the combined feature map to obtain a fused image.
[0027] As a preferred technical solution, in step S1, feature mapping and stitching of hyperspectral remote sensing images and multispectral remote sensing images are achieved through 1×1 convolution.
[0028] Another aspect of the present invention provides a multispectral and hyperspectral image fusion device based on deep separable convolution, comprising:
[0029] An image processing module is used to perform feature mapping processing on the acquired hyperspectral remote sensing images and multispectral remote sensing images to be fused, and obtain input feature maps by splicing;
[0030] A feature extraction module, used to use the input feature map as input to a SwinTransformer network branch for extracting global context features and a separable convolutional layer branch for extracting local features, respectively, to obtain the original feature map output by the two branches and the corresponding downsampled feature map;
[0031] A feature fusion module is used to concatenate the downsampled feature maps corresponding to the two branches and perform transposed convolution processing to obtain an upsampled feature map, combine the upsampled feature map with the original feature map, and use the spatial spectral attention mechanism to achieve adaptive feature fusion to obtain a fused feature map;
[0032] The output module is used to obtain a fused image based on the fused feature map.
[0033] Another aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned multispectral and hyperspectral image fusion method based on deep separable convolution.
[0034] Another aspect of the present invention provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned multispectral and hyperspectral image fusion method based on deep separable convolution.
[0035] Compared with the prior art, the present invention has at least one of the following beneficial effects:
[0036] (1) Realizing high-quality fusion of multispectral and hyperspectral images: In view of the characteristics of multispectral and hyperspectral images, the present invention provides a dual-branch and dual-level multispectral and hyperspectral image fusion model combining Swin Transformer and deep separable convolution, which includes a Swin Transformer network branch for extracting global context features and a separable convolution layer branch for extracting local features. The spatial-spectral feature mapping relationship between the input image pair and the output image is learned and expressed through the two aspects of global and local dependencies, and the spatial-spectral attention mechanism is combined to fuse the extracted features and reconstruct the fused image, thereby achieving high-quality fusion.
[0037] (2) Good fusion effect: For the application scenarios of satellite images, this method has a strong spectral preservation ability while not having obvious chessboard effect, loss of details caused by inconsistent brightness at the edge of the synthetic block, and other poor visual effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 This is a diagram of a multi-spectral and hyper-spectral image fusion network structure based on deep separable multi-modules in an embodiment;
[0039] Figure 2 Schematic diagram of the Swin Transformer network branch in the embodiment;
[0040] Figure 3 This is a schematic diagram of window movement in the embodiment;
[0041] Figure 4 Schematic diagram of calculation of self-attention mechanism in the window after moving in the embodiment;
[0042] Figure 5 This is a schematic diagram of block merging in the embodiment;
[0043] Figure 6 Schematic diagram of the spatial spectral attention mechanism in the embodiment. DETAILED DESCRIPTION
[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0045] Example 1
[0046] In view of the problems existing in the above-mentioned prior art, this embodiment provides a multispectral and hyperspectral image fusion method based on deep separable convolution, introduces the Swin Transformer encoder, uses deep separable convolution in parallel, and uses the spatial spectrum attention mechanism to fuse context information, thereby obtaining a fused high-resolution hyperspectral image with good spectral consistency and rich spatial details. By fusing the spatial spectrum information between different modal data and improving the spatial spectrum resolution of remote sensing images, the information of observed objects can be expressed more comprehensively and accurately, further improving the application capability of remote sensing image data and enhancing its application value.
[0047] See also Figure 1This method provides a dual-branch dual-level multispectral hyperspectral image fusion model that combines Swin Transformer and deep separable convolution, aiming to achieve high-quality high-resolution hyperspectral fusion images. The structure receives two different types of image data, multispectral images and hyperspectral images, and performs preliminary feature mapping and splicing through 1×1 convolution. Subsequently, the feature map is simultaneously input into the Swin Transformer module and the deep separable convolution layer to capture global context and local features, respectively. In the figure, C represents the concat operation, represents concatenation along the channel dimension of the tensor, Respectively represent multiplication.
[0048] Specifically, the model includes the following modules:
[0049] Image processing module, corresponding to Figure 1 The part within the dotted box in the upper middle part is used to perform feature mapping processing on two types of image data respectively, and obtain the input feature map by splicing.
[0050] Feature extraction module, corresponding to Figure 1 The part in the dotted box in the lower left of the figure is used to take the input feature map as the input of the Swin Transformer network branch for extracting global context features and the separable convolutional layer branch for extracting local features, and obtain the original feature maps output by the two branches and the corresponding downsampled feature maps.
[0051] Feature fusion module, corresponding to Figure 1 The part in the lower right dashed box is used to concatenate the downsampled feature maps corresponding to the two branches and perform transposed convolution processing to obtain the upsampled feature map. The upsampled feature map and the original feature map are combined to achieve adaptive feature fusion using the spatial spectral attention mechanism to obtain the fused feature map.
[0052] The output module, not shown in the figure, is used to obtain a fused image based on the fused feature map.
[0053] The process of implementing multispectral and hyperspectral image fusion based on deep separable convolution using the above model includes the following steps:
[0054] Step S1, obtaining the hyperspectral remote sensing image and the multispectral remote sensing image to be fused and performing feature mapping respectively, and obtaining the input feature map by splicing.
[0055] The network receives two different types of image data: hyperspectral remote sensing images and multispectral remote sensing images, and performs preliminary feature mapping and splicing through 1×1 convolution to obtain the input feature map.
[0056] In step S2, the input feature map is used as the input of the Swin Transformer network branch for extracting global context features and the separable convolutional layer branch for extracting local features, respectively, to obtain the original feature maps output by the two branches and the corresponding downsampled feature maps.
[0057] The input feature map is simultaneously input into the Swin Transformer network branch and the depth-wise separable convolutional layer network branch to obtain the feature maps of the corresponding output original sizes. The above two branches are used to capture global context and local features, respectively.
[0058] In the Swin Transformer network branch, the original feature map is further cut into 3×3 segments, and a 3x downsampling rate is used to reduce computational complexity. The depthwise separable convolutional layer network branch further uses a 3×3 convolution kernel and a two-dimensional convolution with a step size of 3 to achieve downsampling, and obtains the downsampled feature maps corresponding to the two branches. The feature maps of the two branches are concatenated to form a two-level feature representation.
[0059] See also Figure 2 Schematic diagram of the Swin Transformer network branch, where (a) is the Swin Transformer module structure, (b) is the Swin Transformer encoder, and the module embedding and linear projection parts are similar to the Vision Transformer. After the convolution layer with a convolution kernel size and a step size equal to the block size is input, the three-dimensional feature map is linearly transformed into a two-dimensional vector through a flattening operation. In the Swin Transformer encoder, the input vector is first calculated for multi-head window self-attention and moving window self-attention to obtain the output feature vector sequence.
[0060] After the output feature vector sequence is inversely transformed to the original size, the image block merging method is used to achieve downsampling, and then it is input into the Swin Transformer module again. The feature vectors of the two outputs are inversely transformed to the input feature size, and feature information at two resolutions is obtained, namely the original feature map and the downsampled feature map of the Swin Transformer branch.
[0061] In the Swin Transformer encoder, a moving window multi-head self-attention mechanism is used to shift the window and calculate the new self-attention score. In the implementation, it is obtained by shifting the feature map. The shift process is as follows: Figure 3 As shown in the figure, according to the displacement step, the corresponding number of rows above the feature map is shifted down, and the number of columns on the left is shifted right.
[0062] In order to make the moving window multi-head self-attention mechanism have an equivalent number of calculation windows, a mask matrix is used to calculate the self-attention score within the window, such as Figure 4 As shown. Taking the lower right corner window as an example, calculate K T A mask matrix is added before the Softmax activation function after Q, so that the corresponding output values of positions that do not belong to the same window are 0. The value of the mask matrix is generally set to -100.
[0063] The calculation process of block merging is as follows: Figure 5 The feature map output by the previous module is inversely transformed to the size before linear flattening of the image block, and the feature map is windowed according to the downsampling ratio 3, divided into several 3×3 blocks, and the pixels at the same position in each window are combined into a new feature map, which is spliced along the channel direction and then downsampled through the fully connected layer. Figure 5 Taking the 6×6×1 feature map as an example, after slicing and merging, a 2×2×9 feature map is obtained, and then a 2×2×3 feature map is finally obtained through the linear layer. The number of channels increases by the number of times the height and width of the feature map are reduced before and after slicing and merging.
[0064] In step S3, the downsampled feature maps corresponding to the two branches are concatenated and transposed convolution is performed to obtain an upsampled feature map. The upsampled feature map and the original feature map are combined, and the spatial spectral attention mechanism is used to realize adaptive feature fusion to obtain a fused feature map.
[0065] Based on the downsampled feature maps corresponding to the two branches, upsampling is performed through concatenation and transposed convolution, and the feature map is restored to the original resolution to obtain the upsampled feature map. Combined with the spatial spectral attention mechanism, the upsampled feature map and the original feature map are combined to achieve adaptive feature fusion and obtain the fused feature map.
[0066] Spatial spectral attention mechanism Figure 6 As shown in the figure, (a) is the channel attention mechanism, (b) is the spatial attention mechanism, and (c) is the spatial-spectral attention mechanism. The channel attention mechanism performs maximum pooling and average pooling on the feature map along the spatial dimension, and then inputs it into the fully connected layer and adds it to obtain the channel attention map through the Sigmoid activation function. The spatial attention mechanism performs the maximum and average operations along the channel dimension to obtain two images. After the convolution layer operation, the output is a feature map with a channel number of 1. The activation function is Sigmoid, and finally the spatial attention map is obtained. The spatial-spectral attention mechanism can effectively improve the efficiency of spatial-spectral feature fusion, improve the accuracy of fused images, filter and screen feature information, and adjust the contribution of different channels or positions to the fused image.
[0067] In the spatial-spectral attention mechanism, the original feature map and the upsampled feature map output by the two branches are first concatenated to obtain the input feature map. The channel attention weight value and the spatial attention weight are calculated using the channel attention and spatial attention, respectively, and the final feature map is obtained by multiplying them.
[0068] Step S4, obtaining a fused image based on the fused feature map.
[0069] Use 1×1 convolution to adjust the number of output channels to obtain the final fused image.
[0070] The above scheme fully considers the characteristics of hyperspectral and high-resolution images, and can improve spatial resolution while retaining spectral information.
[0071] The Swin Transformer captures long-range dependencies and global context, and the deep separable convolution extracts local texture and edge features. The two complement each other to enhance the network's feature expression capabilities. Downsampling and upsampling operations reduce computational complexity and increase the receptive field.
[0072] The spatial-spectral attention mechanism enhances the network’s adaptability and dynamically adjusts the fusion strategy. The final 1×1 convolutional layer integrates and reorganizes features, improves fusion quality, and adapts the network to different numbers of spectral bands.
[0073] This scheme provides an efficient and effective solution for high-resolution hyperspectral image fusion through multi-level and multi-angle feature extraction and fusion strategies, maintains the integrity of spectral information while improving spatial resolution, and provides a high-quality data foundation for remote sensing image analysis and application.
[0074] This solution has the following effects:
[0075] (1) This embodiment provides a dual-branch dual-level multi-spectral and hyperspectral image fusion network that combines Swin Transformer and deep separable convolution. The network learns and expresses the spatial-spectral feature mapping relationship between the input image pair and the output image through global and local dependencies, and uses a spatial-spectral attention module to fuse the extracted features and reconstruct the fused image.
[0076] (2) Experimental results on the Cave dataset and multispectral hyperspectral images of Resource 1-02D show that this scheme can obtain high-resolution hyperspectral fusion images with good spectral consistency and rich spatial details, and the performance of the network model is better than that of classical methods and commonly used deep learning methods.
[0077] (3) The application of this scheme on satellite images shows that while it has a strong spectral preservation ability, there is no obvious chessboard effect, loss of details caused by inconsistent brightness at the edges of the synthetic blocks, and other poor visual effects.
[0078] Example 2
[0079] Based on Example 1, see Figure 1 , this embodiment provides a multispectral and hyperspectral image fusion device based on deep separable convolution, including:
[0080] Image processing module, corresponding to Figure 1 The part within the dotted box in the upper middle part is used to perform feature mapping processing on two types of image data respectively, and obtain the input feature map by splicing.
[0081] Feature extraction module, corresponding to Figure 1 The part in the dotted box in the lower left of the figure is used to take the input feature map as the input of the Swin Transformer network branch for extracting global context features and the separable convolutional layer branch for extracting local features, and obtain the original feature maps output by the two branches and the corresponding downsampled feature maps.
[0082] Feature fusion module, corresponding to Figure 1 The part in the lower right dashed box is used to concatenate the downsampled feature maps corresponding to the two branches and perform transposed convolution processing to obtain the upsampled feature map. The upsampled feature map and the original feature map are combined to achieve adaptive feature fusion using the spatial spectral attention mechanism to obtain the fused feature map.
[0083] The output module, not shown in the figure, is used to obtain a fused image based on the fused feature map.
[0084] Example 3
[0085] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the multispectral and hyperspectral image fusion method based on deep separable convolution as described in Example 1.
[0086] Example 4
[0087] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the multispectral and hyperspectral image fusion method based on deep separable convolution as described in Example 1.
[0088] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A multispectral and hyperspectral image fusion method based on deep separable convolution, It is characterized in that The steps include: Step S1, obtaining a hyperspectral remote sensing image and a multispectral remote sensing image to be fused and performing feature mapping respectively, and obtaining an input feature map by splicing; Step S2, using the input feature map as input to a Swin Transformer network branch for extracting global context features and a separable convolutional layer branch for extracting local features, respectively, to obtain the original feature maps output by the two branches and the corresponding downsampled feature maps; Step S3, concatenating the downsampled feature maps corresponding to the two branches and performing transposed convolution processing to obtain an upsampled feature map, combining the upsampled feature map and the original feature map, and using the spatial spectral attention mechanism to achieve adaptive feature fusion to obtain a fused feature map; Step S4, obtaining a fused image based on the fused feature map.
2. The multispectral and hyperspectral image fusion method based on deep separable convolution according to claim 1, It is characterized in that In step S2, the process of inputting the input feature map into the Swin Transformer network branch to obtain the corresponding original feature map includes the following steps: Step S201, performing block cutting and linear projection processing on the input feature map; Step S202, inputting the feature map processed in step S201 into the Swin Transformer encoder based on window multi-head self-attention and moving window multi-head sub-attention to obtain the original feature map of the Swin Transformer network branch; In step S203, the original feature map of the Swin Transformer network branch is inversely transformed and the image blocks are merged to achieve downsampling, and then input again into the Swin Transformer encoder based on window multi-head self-attention and moving window multi-head sub-attention to obtain the downsampled feature map of the Swin Transformer network branch.
3. The multispectral and hyperspectral image fusion method based on deep separable convolution according to claim 1, It is characterized in that In step S3, the process of combining the upsampled feature map and the original feature map and implementing adaptive feature fusion using the spatial spectrum attention mechanism to obtain the fused features includes the following steps: Step S301, concatenating the original feature maps output by the two branches and the upsampled feature map to obtain a first feature map; Step S302, based on the first feature map, using channel attention calculation to obtain a channel attention map; Step S303, concatenating the channel attention map and the first feature map to obtain a second feature map, and calculating a spatial attention map based on the second feature map using a spatial attention mechanism; Step S304: merge the second feature map and the spatial attention map to obtain a fused feature map.
4. The multispectral and hyperspectral image fusion method based on deep separable convolution according to claim 3, It is characterized in that In step S302, the process of obtaining the channel attention map by using the channel attention calculation includes: The feature map is processed by maximum pooling and average pooling along the spatial dimension, input into the fully connected layer, and added and calculated by the activation function to obtain the channel attention map.
5. The multispectral and hyperspectral image fusion method based on deep separable convolution according to claim 3, It is characterized in that In step S303, the process of calculating the spatial attention map using the spatial attention mechanism includes: The two images are obtained by taking the maximum and average operations along the channel dimension. After the convolution layer operation, the output is a feature map with a channel number of 1, and after the activation function, the spatial attention map is obtained.
6. The multispectral and hyperspectral image fusion method based on deep separable convolution according to claim 1, It is characterized in that In the step S4, the number of output channels of the combined feature map is adjusted using 1×1 convolution to obtain a fused image.
7. The multispectral and hyperspectral image fusion method based on deep separable convolution according to claim 1, It is characterized in that In the step S1, feature mapping and stitching of hyperspectral remote sensing images and multispectral remote sensing images are achieved through 1×1 convolution.
8. A multispectral and hyperspectral image fusion device based on deep separable convolution, It is characterized in that include: An image processing module is used to perform feature mapping processing on the acquired hyperspectral remote sensing images and multispectral remote sensing images to be fused, and obtain input feature maps by splicing; A feature extraction module, used to use the input feature map as input to a SwinTransformer network branch for extracting global context features and a separable convolutional layer branch for extracting local features, respectively, to obtain the original feature map output by the two branches and the corresponding downsampled feature map; A feature fusion module is used to concatenate the downsampled feature maps corresponding to the two branches and perform transposed convolution processing to obtain an upsampled feature map, combine the upsampled feature map with the original feature map, and use the spatial spectral attention mechanism to achieve adaptive feature fusion to obtain a fused feature map; The output module is used to obtain a fused image based on the fused feature map.
9. An electronic device, It is characterized in that include: One or more processors and a memory, wherein the memory stores one or more programs, wherein the one or more programs include instructions for executing the multispectral and hyperspectral image fusion method based on deep separable convolution as described in any one of claims 1-7.
10. A computer-readable storage medium, It is characterized in that It includes one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the multispectral and hyperspectral image fusion method based on deep separable convolution as described in any one of claims 1-7.
Citation Information
Cited By
Remote sensing image fusion method based on local-global spatial-spectral correlation prior
CN120147810A
Hyperspectral image demixing method and device, electronic equipment and computer program product
CN121010897A
