Remote sensing multispectral image fusion method and system based on multi-stage feature correction

Abstract

The invention relates to the field of computer image processing, and provides a remote sensing multispectral image fusion method and system based on multi-stage feature correction, and the method comprises the steps: obtaining a spectrum mask through a low-resolution multispectral image and a panchromatic image, and covering the panchromatic image through the spectrum mask to obtain a corrected panchromatic image; performing scale feature enhancement on the low-resolution multispectral image, and performing down-sampling and convolution on the corrected panchromatic image to obtain image scale features; obtaining attention block input, and mapping the attention block input to obtain a mapping vector group; performing image correction by using the mapping vector group to obtain a feature correction vector, performing iteration on the feature correction vector to obtain a target feature correction vector, and obtaining a fusion feature matrix; and iterating the fusion feature matrix to obtain target fusion features, and carrying out feature refinement and convolution on the target fusion features layer by layer to obtain a high-resolution multispectral image. According to the invention, a high-fidelity and high-resolution multispectral image can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and particularly to a remote sensing multispectral image fusion method and system based on multi-stage feature correction. Background Art

[0002] Multispectral sensors cover common visible light and some near-infrared bands and can resolve specific spectral features, so they are widely used in remote sensing observation satellites. However, the spatial resolution of images captured by multispectral sensors is restricted by physical performance and current technological conditions, and the performance in spatial resolution is not good. To make up for this defect, satellites usually carry panchromatic sensors to capture and record single-band panchromatic images with higher spatial resolution. Although panchromatic images can significantly enhance spatial details by integrating optical signals in a wide band, their single-band characteristics lead to a serious lack of spectral information and it is difficult to meet the application requirements that rely on multispectral feature analysis such as land cover classification and vegetation health diagnosis. Therefore, it is necessary to meet these requirements through multispectral panchromatic sharpening technology, whose goal is to fuse the high-frequency texture information of panchromatic images with the spectral features of low-spatial-resolution multispectral images through methods to generate high-resolution multispectral images with both high spatial resolution and multispectral fidelity. This technology has received considerable attention in the field of remote sensing and is of great significance in various remote sensing tasks such as remote sensing mapping, resource investigation, and classification.

[0003] Multispectral panchromatic sharpening methods are mainly divided into two categories: traditional methods and deep learning-based methods. Traditional methods can be divided into methods based on component replacement, methods based on multi-resolution analysis, and methods based on variational optimization. However, due to limited feature representation and a strong dependence on linear mathematical modeling, these methods often suffer from significant spectral and spatial distortions and impose a high computational burden, which limits their practical applications.

[0004] In recent years, due to deep learning technology, especially the powerful feature extraction ability of convolutional neural networks, there are more and more deep learning-based multispectral panchromatic sharpening methods and they have achieved better results than traditional methods. However, since existing methods ignore the impact of the modality difference between the two input images on feature fusion, spectral and spatial distortions will occur in the fused images. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the related art. For this purpose, the present invention provides a remote sensing multispectral image fusion method and system based on multi-stage feature correction to achieve the generation of high-resolution multispectral images with both high spatial resolution and multispectral fidelity.

[0006] The present invention provides a remote sensing multispectral image fusion method based on multi-stage feature correction, including: S1: Obtain a low - resolution multispectral image and a panchromatic image, obtain a spectral mask from the low - resolution multispectral image and the panchromatic image, and use the spectral mask to cover the panchromatic image to obtain a corrected panchromatic image; S2: Perform scale - feature enhancement on the low - resolution multispectral image through a multi - scale channel attention block, and perform downsampling and convolution on the corrected panchromatic image to obtain image scale features; S3: Obtain the input of the attention block from the image scale features, map the input of the attention block, and obtain a mapped vector group including a panchromatic vector group and a low - resolution vector group; S4: Use the mapped vector group for image correction to obtain a feature correction vector, iterate the feature correction vector to obtain a target feature correction vector, and input the target feature correction vector into a feed - forward neural network to obtain a fused feature matrix; S5: Repeat S3 to S4 to iterate the fused feature matrix to obtain a target fused feature, construct a feature refinement block, and perform layer - by - layer feature fusion and convolution on the target fused feature through the feature refinement block to obtain a high - resolution multispectral image.

[0007] According to the remote - sensing multispectral image fusion method based on multi - stage feature correction provided by the present invention, in step S1, the low - resolution multispectral image is upsampled to obtain an upsampled multispectral image, local entropy operation is performed on the upsampled multispectral image, and guided filtering operation and normalization are performed on the upsampled multispectral image after local entropy operation and the panchromatic image to obtain a spectral mask.

[0008] According to the remote - sensing multispectral image fusion method based on multi - stage feature correction provided by the present invention, in step S1, the spectral mask is multiplied point - by - point with the panchromatic image to complete the covering of the panchromatic image.

[0009] According to the remote - sensing multispectral image fusion method based on multi - stage feature correction provided by the present invention, in step S2, the input of the multi - scale channel attention block is obtained from the low - resolution multispectral image. In the multi - scale channel attention block, multi - scale convolution and channel attention operation are performed on the input of the multi - scale channel attention block to obtain low - resolution scale features.

[0010] According to the remote - sensing multispectral image fusion method based on multi - stage feature correction provided by the present invention, in step S2, convolution and downsampling are performed on the corrected panchromatic image to obtain panchromatic scale features, and the image scale features are obtained through the panchromatic scale features and the low - resolution scale features.

[0011] According to the remote - sensing multispectral image fusion method based on multi - stage feature correction provided by the present invention, in step S3, the input of the attention block is mapped into a Transformer structure to obtain a panchromatic vector group and a low - resolution vector group respectively.

[0012] According to the remote sensing multispectral image fusion method based on multi-stage feature correction provided by the present invention, in step S4, a deformation operation and normalization are performed on the mapping vector group to obtain a vertical matrix and a horizontal matrix. A feature correction vector including a panchromatic feature correction vector and a low-spectral feature correction vector is obtained through the vertical matrix and the horizontal matrix. The low-spectral feature correction vector and the panchromatic feature correction vector are used as inputs to the attention block and the feature correction vector is iterated to obtain a target feature correction vector.

[0013] According to the remote sensing multispectral image fusion method based on multi-stage feature correction provided by the present invention, in step S4, the target feature correction vector is input into a feedforward neural network, and convolution, normalization, and residual operations are performed on the target feature correction vector in the feedforward neural network to obtain the fusion feature matrix.

[0014] According to the remote sensing multispectral image fusion method based on multi-stage feature correction provided by the present invention, in step S5, a refined feature is obtained through the target fusion feature. The refined feature is iterated through the feature refinement block and the target fusion feature, and layer-by-layer feature fusion and convolution are performed on the target fusion feature to obtain a target refined feature. Feature refinement and convolution are performed on the target refined feature to obtain the high-resolution multispectral image.

[0015] The present invention also provides a remote sensing multispectral image fusion system based on multi-stage feature correction, including: Panchromatic image correction module: used to obtain a low-resolution multispectral image and a panchromatic image, obtain a spectral mask through the low-resolution multispectral image and the panchromatic image, and use the spectral mask to cover the panchromatic image to obtain a corrected panchromatic image; Image scale feature module: used to enhance the scale features of the low-resolution multispectral image through a multi-scale channel attention block, and perform downsampling and convolution on the corrected panchromatic image to obtain image scale features; Mapping vector group module: used to obtain an input to the attention block through the image scale features, and map the input to the attention block to obtain a mapping vector group including a panchromatic vector group and a low-resolution vector group; Fusion feature matrix module: used to perform image correction using the mapping vector group to obtain a feature correction vector, iterate the feature correction vector to obtain a target feature correction vector, and input the target feature correction vector into a feedforward neural network to obtain a fusion feature matrix; High-resolution multispectral image module: used to repeat the mapping vector group module to the fusion feature matrix module to iterate the fusion feature matrix to obtain a target fusion feature, construct a feature refinement block, and perform layer-by-layer feature fusion and convolution on the target fusion feature through the feature refinement block to obtain a high-resolution multispectral image.

[0016] One or more of the above technical solutions in the embodiments of the present invention have at least one of the following technical effects: The remote sensing multispectral image fusion method and system based on multi-stage feature correction provided by the present invention effectively extract and enhance the features of the low-resolution multispectral image through the multi-scale channel attention block, and in the process of obtaining the target fusion features, continuous sparse attention is used to replace single-layer dense attention, realizing lightweight calculation, reducing the use of memory, establishing the relationship between the panchromatic image and the low-resolution multispectral image, thereby correcting the difference between the panchromatic image and the low-resolution multispectral image, reducing the distortion and aberration caused by information interference in the fusion process, and thus generating a high-resolution multispectral image with less distortion.

[0017] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. Description of the Drawings

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0019] Figure 1 It is a schematic flowchart of the remote sensing multispectral image fusion method based on multi-stage feature correction provided by the present invention.

[0020] Figure 2 It is a comparison diagram of the mean absolute error map of the remote sensing multispectral image fusion method based on multi-stage feature correction provided by the present invention.

[0021] Figure 3 It is a schematic structural diagram of the remote sensing multispectral image fusion based on multi-stage feature correction provided by the present invention.

[0022] Reference Signs: 100, corrected panchromatic image module; 200, image scale feature module; 300, mapping vector group module; 400, fusion feature matrix module; 500, high-resolution multispectral image module. Detailed Embodiments

[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention. The following embodiments are used to illustrate the present invention but cannot be used to limit the scope of the present invention.

[0024] In the description of the embodiments of the present invention, it should be noted that the terms "first", "second", and "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0025] In the description of this specification, the descriptions with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the embodiments of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0026] The following Figures 1 to 3 describes the implementation scheme of the present invention. Figure 1 is a schematic flowchart of a remote sensing multispectral image fusion method based on multi-stage feature correction provided by the present invention. As shown in the figure, first, a spectral mask is obtained, and the panchromatic image is covered with the spectral mask to obtain a corrected panchromatic image; then, the scale features of the low-resolution multispectral image are enhanced, and the corrected panchromatic image is downsampled and convolved to obtain image scale features; subsequently, the input of the attention block is obtained, and the input of the attention block is mapped to obtain a mapped vector group including a panchromatic vector group and a low-resolution vector group; then, image correction is performed to obtain a feature correction vector, and the target feature correction vector is obtained and input into the feedforward neural network to obtain a fusion feature matrix; finally, the target fusion feature is obtained, and the target fusion feature is subjected to layer-by-layer feature refinement and convolution to obtain a high-resolution multispectral image.

[0027] The present invention provides a remote sensing multispectral image fusion method based on multi-stage feature correction, specifically including: S1: Obtain a low-resolution multispectral image and a panchromatic image, obtain a spectral mask from the low-resolution multispectral image and the panchromatic image, and cover the panchromatic image with the spectral mask to obtain a corrected panchromatic image; Further, the purpose of this stage is to obtain a spectral mask, so as to cover the panchromatic image with the spectral mask to obtain a corrected panchromatic image. Specifically, in step S1, the low-resolution multispectral image is upsampled to obtain an upsampled multispectral image. The local entropy operation is performed on the upsampled multispectral image, and the guided filtering operation and normalization are performed on the upsampled multispectral image and the panchromatic image after the local entropy operation to obtain a spectral mask.

[0028] In step S1, the spectral mask is multiplied by the panchromatic image point by point to complete the covering of the panchromatic image.

[0029] For the above steps, the specific implementation in this embodiment is as follows: First, obtain the low-resolution multispectral image taken by the remote sensing satellite and the panchromatic image . Then, the low-resolution multispectral image is upsampled to obtain an upsampled multispectral image . Subsequently, the local entropy operation is performed on the upsampled multispectral image, and the guided filtering operation and normalization are performed on the upsampled multispectral image and the panchromatic image after the local entropy operation to obtain a spectral mask SM: Among them, represents performing the local entropy operation on the content in the brackets, represents performing the guided filtering operation on the content in the brackets, and Norm() represents performing the minimum-maximum normalization on the content in the brackets.

[0030] Finally, the spectral mask is multiplied by the panchromatic image point by point to complete the covering of the panchromatic image, and a corrected panchromatic image with the same number of bands as the low-resolution multispectral image is obtained : Among them, represents point-by-point multiplication.

[0031] S2: Perform scale feature enhancement on the low-resolution multispectral image through a multi-scale channel attention block, and perform downsampling and convolution on the corrected panchromatic image to obtain image scale features; Further, this stage is to obtain a corrected panchromatic image through multi-scale channel attention, downsampling, and convolution. Specifically, in step S2, the input of the multi-scale channel attention block is obtained from the low-resolution multispectral image. In the multi-scale channel attention block, multi-scale convolution and channel attention operations are performed on the input of the multi-scale channel attention block to obtain low-resolution scale features.

[0032] In step S2, the corrected panchromatic image is convolved and downsampled to obtain panchromatic scale features, and the image scale features are obtained from the panchromatic scale features and the low-resolution scale features.

[0033] For the above steps, the specific implementation in this embodiment is as follows: First, determine the number of multi-scale channel attention blocks. In this embodiment, the number of multi-scale channel attention blocks is 3 and they are connected in sequence. Subsequently, for the first multi-scale channel attention block, the low-resolution multispectral image is convolved with a 3×3 filter to obtain the input of the multi-scale channel attention block of the first multi-scale channel attention block , and for the other multi-scale channel attention blocks, the low-resolution scale features of the previous multi-scale channel attention block after upsampling are convolved with a 3×3 filter and used as the input of the multi-scale channel attention block.

[0034] In the multi-scale channel attention block, first perform multi-scale convolution on the input of the multi-scale channel attention block to obtain the multi-scale convolution output of the k-th scale in the s-th multi-scale channel attention block : where denotes performing a 1×1 convolution on the content within the parentheses, denotes performing a k×k convolution on the content within the parentheses, where k takes values of 3, 5, and 7. Then, perform channel attention operation to obtain the s-th low-resolution scale feature : where Cat() denotes concatenating the content within the parentheses, and CA() denotes performing channel attention operation on the content within the parentheses.

[0035] In this way, the s-th low-resolution scale feature obtained for the s-th multi-scale channel attention block can be expressed as: where s is an integer, Up() denotes performing an upsampling operation on the content within the parentheses, MCAB() denotes inputting the content within the parentheses into the multi-scale channel attention block, denotes performing a 3×3 convolution on the content within the parentheses.

[0036] Then, convolve and downsample the corrected panchromatic image, and determine the number of times of convolution and downsampling according to the number of multi-scale channel attention blocks to obtain the s-th panchromatic scale feature : Among them, Down() represents downsampling the content within the parentheses, and the value of s starts from 3 and decreases by 1 each time a convolution and downsampling are performed. All the panchromatic scale features and low-resolution scale features are used as image scale features.

[0037] S3: Obtain the input of the attention block from the image scale features, and map the input of the attention block to obtain a mapping vector group including a panchromatic vector group and a low-resolution vector group; Furthermore, the purpose of this stage is to obtain the input of the attention block and perform mapping to obtain a mapping vector group. Specifically, in step S3, the input of the attention block is mapped into the Transformer structure to obtain a panchromatic vector group and a low-resolution vector group respectively.

[0038] For the above steps, the specific implementation in this embodiment is as follows: First, establish multiple feature cross-attention blocks according to the number of panchromatic scale features and low-resolution scale features. Since the number of both panchromatic scale features and low-resolution scale features is 3, the number of feature cross-attention blocks is also 3. Take the 1st panchromatic scale feature and the 1st low-resolution scale feature as the input of the 1st feature cross-attention block, that is, the input of the attention block. Then, let the input of the attention block including the 1st panchromatic scale feature and the 1st low-resolution scale feature enter the 1st multi-scale channel attention block, and map it into the Transformer structure in the multi-scale channel attention block, and then a panchromatic vector group and a low-resolution vector group can be obtained respectively, thus forming a mapping vector group: Among them, LN() represents performing layer normalization operation of the Transformer structure on the content within the parentheses, is the input of the sth panchromatic attention block, which is the 1st panchromatic scale feature here; is the input of the sth low-resolution attention block, which is the 1st low-resolution scale feature here; is the first panchromatic vector, is the second panchromatic vector, is the third panchromatic vector, is the first low-resolution vector, is the second low-resolution vector, is the third low-resolution vector.

[0039] S4: Use the mapping vector group to perform image correction to obtain a feature correction vector, iterate the feature correction vector to obtain a target feature correction vector, and input the target feature correction vector into a feed-forward neural network to obtain a fused feature matrix; Further, the purpose of this stage is to use the mapping vector group for image correction and iterate the feature correction vector to obtain the target feature correction vector, and finally output the fused feature matrix. Specifically, in step S4, the mapping vector group is deformed and normalized to obtain a vertical matrix and a horizontal matrix. The feature correction vector including the panchromatic feature correction vector and the low-spectral feature correction vector is obtained through the vertical matrix and the horizontal matrix. The low-spectral feature correction vector and the panchromatic feature correction vector are used as the input of the attention block and the feature correction vector is iterated to obtain the target feature correction vector.

[0040] In step S4, the target feature correction vector is input into the feed-forward neural network, and the target feature correction vector is subjected to convolution, normalization, and residual operations in the feed-forward neural network to obtain the fused feature matrix.

[0041] For the above steps, the specific implementation in this embodiment is as follows: In the feature cross-attention block, during the process of deforming and normalizing the mapping vector group, first, the first low-resolution vector and the second panchromatic vector are respectively subjected to deformation operations and normalization in the horizontal dimension and the vertical dimension to obtain the first vertical matrix of the s-th feature cross-attention block and the first horizontal matrix of the s-th feature cross-attention block : wherein, represents the deformation operation in the vertical dimension, represents the deformation operation in the vertical dimension, softmax() represents performing the softmax transformation on the content in the parentheses, that is, normalizing using the exponential function, T represents the transpose, and × represents the multiplication operation. Then, the first panchromatic vector and the second low-resolution vector are respectively subjected to deformation operations and normalization in the horizontal dimension and the vertical dimension to obtain the second vertical matrix of the s-th feature cross-attention block and the second horizontal matrix of the s-th feature cross-attention block : The first vertical matrix and the second vertical matrix are used as the vertical matrix, and the first horizontal matrix and the second horizontal matrix are used as the horizontal matrix, thus completing the deformation operation and normalization of the mapping vector group. Subsequently, the panchromatic feature correction vector and the low-spectral feature correction vector of the s-th feature cross-attention block are obtained through the vertical matrix and the horizontal matrix, and are used as the feature correction vector of the s-th feature cross-attention block: Subsequently, iterative operations are performed on the feature correction vectors, that is, the low-spectral feature correction vector and the panchromatic feature correction vector are respectively used as the input of the attention block and mapped into the Transformer structure to obtain a new group of panchromatic vectors and a new group of low-resolution vectors. Finally, a new low-spectral feature correction vector and a new panchromatic feature correction vector are obtained. According to experience, the number of iterations R is determined and this process is executed R times to obtain the target panchromatic feature correction vector of the s-th feature cross-attention block. and the target low-resolution feature vector of the s-th feature cross-attention block , and the fusion correction vector of the s-th feature cross-attention block is obtained. .

[0042] Subsequently, the target panchromatic feature correction vector, the target low-resolution feature vector, and the fusion correction vector are used as the target feature correction vector, and the target feature correction vector is input into the feed-forward neural network. In the feed-forward neural network, convolution, normalization, and residual operations can be performed on the target feature correction vector to obtain the fusion feature matrix of the s-th feature cross-attention block and output it from the feature cross-attention block: where FFN() represents performing convolution, normalization, and residual operations on the content in the parentheses on the target feature correction vector through the feed-forward neural network. is the multi-spectral fusion feature of the s-th feature cross-attention block, is the panchromatic correction feature of the s-th feature cross-attention block, is the fusion feature of the s-th feature cross-attention block.

[0043] S5: Repeat S3 to S4 to perform iterative operations on the fusion feature matrix to obtain the target fusion feature, construct a feature refinement block, and perform layer-by-layer feature fusion and convolution on the target fusion feature through the feature refinement block to obtain a high-resolution multi-spectral image.

[0044] Furthermore, the purpose of this stage is to obtain the target fusion feature, construct a feature refinement block, and use the feature refinement block to obtain a high-resolution multi-spectral image. Specifically, in step S5, the refined feature is obtained through the target fusion feature, and the refined feature is iterated through the feature refinement block and the target fusion feature, and layer-by-layer feature fusion and convolution are performed on the target fusion feature to obtain the target refined feature. Feature refinement and convolution are performed on the target refined feature to obtain the high-resolution multi-spectral image.

[0045] For the above steps, the specific implementation in this embodiment is as follows: First, repeat steps S3 to S4 to iterate on the fused feature matrix. In this embodiment, this is to obtain the panchromatic attention block input and the low-resolution attention block input of the next feature cross-attention block through the fused feature matrix and the image scale feature: And repeat steps S3 to S4, that is, obtain a new fused feature matrix in the second and third feature cross-attention blocks, and use the fused features output by each feature cross-attention block as the target fused feature.

[0046] Next, construct a feature refinement block to perform layer-by-layer feature fusion and convolution on the target fused feature. Since in this embodiment, the target fused feature contains a total of three fused features, the process of layer-by-layer feature fusion is also divided into three layers, and each layer can output a refined feature of the s-th layer : Among them, represents the fused feature of the first feature cross-attention block, MFRB() represents inputting the content in the parentheses into the feature refinement block, Ps() represents performing a PixShuffle operation on the content in the parentheses. The PixShuffle operation can refine the image, and splicing the output of the refined feature refinement block with can complete the feature fusion of . Among them, in the feature refinement block, for each input, there is: Among them, is the first feature refinement branch, is the second feature refinement branch, Linear() represents performing a full connection on the content in the parentheses through a fully connected layer, DWConv() represents performing a depth convolution on the content in the parentheses, SiLU() represents activating the content in the parentheses through an s-shaped weighted linear unit activation function, ESS2D() represents performing an ESS2D mechanism operation on the content in the parentheses, and α is a learnable scaling factor. is the output of the feature refinement block.

[0047] Iterate on the refined features layer by layer to complete the layer-by-layer fusion and convolution of the target fused feature, and use the refined feature of the last layer, which is the 3rd layer in this embodiment, as the target refined feature , perform feature refinement and convolution on the target refined feature again, and a high-resolution multispectral image can be obtained : To verify the effectiveness of the present invention, the method provided by the present invention was compared with the high-resolution multispectral images output by other methods on the public dataset WV3, and a comparison graph of the average absolute error graphs of the high-resolution multispectral images output by the present method and the method provided by the present invention and other methods was obtained, as Figure 2 shown, where LAGNet, BiMPAN, PSCF-Net, and SSMNet are all existing methods. It can be seen that there are more dark blue areas in the average absolute error graph of the method provided by the present invention, indicating that the fusion error is smaller. It can be seen that the method provided by the present invention has achieved the best resolution and the least distortion. The specific experimental results are compared as shown in Table 1: Table 1 Performance comparison table of high-resolution multispectral images obtained by the method provided by the present invention and other methods

[0048] Among them, SSIM is the structural similarity, SCC is the spectral cross-correlation, SAM is the spectral angle mapping, RMSE is the root mean square error, ERGAS is the dimensionless global relative error, PSNR is the peak signal-to-noise ratio. An upward arrow indicates that the higher the value, the better, and a downward arrow indicates that the lower the value, the better. The underline indicates that the value of this method has achieved a sub-optimal performance in this item, and the bold indicates that the value of this method has achieved the optimal performance in this item. It can be seen that the method provided by the present invention has achieved the best performance in most values.

[0049] Next, a remote sensing multispectral image fusion device based on multi-stage feature correction provided by the present invention will be described. The remote sensing multispectral image fusion device based on multi-stage feature correction described below can be mutually referred to the remote sensing multispectral image fusion method based on multi-stage feature correction described above.

[0050] Figure 3 The structural schematic diagram of a remote sensing multispectral image fusion system based on multi-stage feature correction is exemplified, as Figure 3 shown, which is used to execute the remote sensing multispectral image fusion method based on multi-stage feature correction as described above, including: Panchromatic image correction module 100: used to obtain a low-resolution multispectral image and a panchromatic image, obtain a spectral mask through the low-resolution multispectral image and the panchromatic image, and use the spectral mask to cover the panchromatic image to obtain a corrected panchromatic image; Image scale feature module 200: used to enhance the scale features of the low-resolution multispectral image through a multi-scale channel attention block, and perform downsampling and convolution on the corrected panchromatic image to obtain image scale features; Mapping vector group module 300: It is used to obtain the input of the attention block through the image scale features, map the input of the attention block, and obtain a mapping vector group including a panchromatic vector group and a low-resolution vector group; Fusion feature matrix module 400: It is used to perform image correction using the mapping vector group to obtain a feature correction vector, iterate the feature correction vector to obtain a target feature correction vector, and input the target feature correction vector into a feedforward neural network to obtain a fusion feature matrix; High-resolution multispectral image module 500: It is used to repeat the mapping vector group module to the fusion feature matrix module to iterate the fusion feature matrix to obtain a target fusion feature, construct a feature refinement block, and perform layer-by-layer feature fusion and convolution on the target fusion feature through the feature refinement block to obtain a high-resolution multispectral image.

[0051] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0052] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solutions or the part that contributes to the prior art can be embodied in the form of a software product, and this computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.

Claims

1. A remote sensing multispectral image fusion method based on multi-stage feature correction, characterized in that: include: S1: Obtain a low-resolution multispectral image and a panchromatic image, obtain a spectral mask through the low-resolution multispectral image and the panchromatic image, use the spectral mask to cover the panchromatic image, and obtain a corrected panchromatic image; S2: The scale feature of the low-resolution multispectral image is enhanced through the multi-scale channel attention block, and the rectified panchromatic image is downsampled and convolved to obtain the image scale feature; S3: Obtain the attention block input through the image scale feature, map the attention block input, and obtain a mapping vector group including a full-color vector group and a low-resolution vector group; S4: Use the mapping vector group to perform image correction to obtain a feature correction vector, iterate the feature correction vector to obtain a target feature correction vector, input the target feature correction vector into a feedforward neural network to obtain a fusion feature matrix; S5: Repeat S3 to S4 to iterate the fusion feature matrix to obtain the target fusion feature, construct a feature refinement block, and perform layer-by-layer feature fusion and convolution on the target fusion feature through the feature refinement block to obtain a high-resolution multispectral image.

2. The remote sensing multispectral image fusion method based on multi-stage feature correction according to claim 1 is characterized in that: In step S1, the low-resolution multispectral image is upsampled to obtain an upsampled multispectral image, a local entropy operation is performed on the upsampled multispectral image, and a guided filtering operation and normalization are performed on the upsampled multispectral image and the panchromatic image after the local entropy operation to obtain a spectral mask.

3. The remote sensing multispectral image fusion method based on multi-stage feature correction according to claim 1 is characterized in that: In step S1, the spectral mask is multiplied point by point with the full-color image to complete the masking of the full-color image.

4. The remote sensing multispectral image fusion method based on multi-stage feature correction according to claim 1 is characterized in that: In step S2, a multi-scale channel attention block input is obtained through a low-resolution multispectral image. In the multi-scale channel attention block, multi-scale convolution and channel attention operations are performed on the multi-scale channel attention block input to obtain a low-resolution scale feature.

5. The remote sensing multispectral image fusion method based on multi-stage feature correction according to claim 4 is characterized in that: In step S2, the corrected full-color image is convolved and down-sampled to obtain a full-color scale feature, and the image scale feature is obtained through the full-color scale feature and the low-resolution scale feature.

6. The remote sensing multispectral image fusion method based on multi-stage feature correction according to claim 1 is characterized in that: In step S3, the attention block input is mapped to the Transformer structure to obtain a full-color vector group and a low-resolution vector group respectively.

7. The remote sensing multispectral image fusion method based on multi-stage feature correction according to claim 1 is characterized in that: In step S4, the mapping vector group is deformed and normalized to obtain a vertical matrix and a horizontal matrix. A feature correction vector including a full-color feature correction vector and a low-spectral feature correction vector is obtained through the vertical matrix and the horizontal matrix. The low-spectral feature correction vector and the full-color feature correction vector are used as attention block inputs and the feature correction vectors are iterated to obtain a target feature correction vector.

8. The remote sensing multispectral image fusion method based on multi-stage feature correction according to claim 1 is characterized in that: In step S4, the target feature correction vector is input into a feedforward neural network, and convolution, normalization and residual operation are performed on the target feature correction vector in the feedforward neural network to obtain the fusion feature matrix.

9. The remote sensing multispectral image fusion method based on multi-stage feature correction according to claim 1 is characterized in that: In step S5, a refined feature is obtained through the target fusion feature, the refined feature is iterated through the feature refinement block and the target fusion feature, and the target fusion feature is subjected to layer-by-layer feature fusion and convolution to obtain a target refined feature, and the target refined feature is subjected to feature refinement and convolution to obtain the high-resolution multispectral image.

10. A remote sensing multispectral image fusion system based on multi-stage feature correction, used to execute the remote sensing multispectral image fusion method based on multi-stage feature correction as claimed in any one of claims 1 to 9, characterized in that: include: Corrected panchromatic image module: used to obtain low-resolution multispectral images and panchromatic images, obtain spectral masks through low-resolution multispectral images and panchromatic images, use the spectral masks to cover the panchromatic image, and obtain a corrected panchromatic image; Image scale feature module: used to enhance the scale features of low-resolution multispectral images through multi-scale channel attention blocks, and downsample and convolve the rectified panchromatic image to obtain image scale features; Mapping vector group module: used to obtain the attention block input through image scale features, map the attention block input, and obtain a mapping vector group including a full-color vector group and a low-resolution vector group; Fusion feature matrix module: used to use the mapping vector group to perform image correction, obtain the feature correction vector, iterate the feature correction vector to obtain the target feature correction vector, input the target feature correction vector into the feedforward neural network, and obtain the fusion feature matrix; High-resolution multispectral image module: It is used to repeatedly map the vector group module to the fusion feature matrix module to iterate the fusion feature matrix, obtain the target fusion feature, construct the feature refinement block, and perform layer-by-layer feature fusion and convolution on the target fusion feature through the feature refinement block to obtain a high-resolution multispectral image.

Images