Multispectral and hyperspectral image fusion method based on spatial frequency cooperative network

Abstract

The application discloses a multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network, relates to the technical field of remote sensing, and discloses a multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network, which comprises the following steps: acquiring a multispectral image and a hyperspectral image of a remote sensing image; based on a CSAM module in a preset fusion model, capturing a spatial dependence relationship of spatial branch features of the multispectral image and the hyperspectral image to obtain spatial features; based on an SFBM module in the preset fusion model, determining frequency dependence of the spatial branch features to obtain frequency features; and based on a convolution layer in the preset fusion model, fusing the spatial features and the frequency features to obtain a fusion image of the multispectral image and the hyperspectral image. Through integration of spatial domain and frequency domain features, the model can more accurately capture and fuse complex information in the multispectral and hyperspectral images, and the quality of image fusion is significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing, in particular to a multi-spectral and hyperspectral image fusion method based on a spatial frequency collaborative network. BACKGROUND

[0002] Hyperspectral images can provide hundreds to thousands of narrow-band spectral information, capturing the unique characteristics of various materials, and multispectral imaging systems provide high spatial resolution. In order to generate high spatial resolution hyperspectral images, the fusion of multispectral and hyperspectral images combines the advantages of both.

[0003] Since the frequency domain features can provide a global view, capture the global correlation in the image, effectively supplement the local features of the spatial domain, and thus enhance the overall representation ability of the image, at present, by utilizing directional pair multi-head cross attention to capture the interaction between different modalities, thereby promoting the information transmission between modalities, a SwinTransformer block is added after the cross attention to enhance the self-attention in the context to better fuse multispectral images and hyperspectral images. It can also replace the self-attention in vision through Fourier transform and a learnable filter, exchange the amplitude and phase components of the image in the Fourier frequency domain to enhance and adjust the frequency information, and realize long spatial-temporal dependence in the frequency domain. The image is decomposed into amplitude and phase components to effectively describe the structural information of the image. Although Fourier transform has advantages in enhancing the frequency domain information capturing ability, it cannot effectively integrate frequency domain and spatial features, thereby limiting the quality of image fusion.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a multi-spectral and hyperspectral image fusion method based on a spatial frequency collaborative network, which aims to solve the technical problem that the frequency domain and spatial features cannot be effectively integrated, thereby limiting the quality of image fusion.

[0006] To achieve the above purpose, the present application provides a multi-spectral and hyperspectral image fusion method based on a spatial frequency collaborative network, which comprises:

[0007] Obtaining a multispectral image and a hyperspectral image of a remote sensing image;

[0008] Based on the CSAM module in the preset fusion model, the spatial dependence relationship of the spatial branch features of the multispectral image and the hyperspectral image is captured to obtain spatial features, and the spatial branch features are features after spatial branch processing of the multispectral image and the hyperspectral image;

[0009] determine the spatial branch features based on the SFBM module in the preset fusion model to obtain frequency features;

[0010] fuse the spatial features and the frequency features based on the convolution layer in the preset fusion model to obtain a fusion image of the multispectral image and the hyperspectral image.

[0011] In an embodiment, the step of obtaining the spatial features based on the CSAM module in the preset fusion model and capturing the spatial dependency relationship of the spatial branch features of the multispectral image and the hyperspectral image comprises:

[0012] perform spatial branch processing on the spliced image of the multispectral image and the hyperspectral image based on the convolution layer in the preset fusion model to obtain spatial branch features of the spliced image;

[0013] compress the spatial branch features based on the CSAM module in the preset fusion model to obtain a feature representation map of the spatial branch features;

[0014] perform the self-attention mechanism in the CSAM module based on the feature representation map to capture the spatial dependency relationship of the spatial branch features to obtain the spatial features.

[0015] In an embodiment, the step of compressing the spatial branch features based on the CSAM module in the preset fusion model to obtain a feature representation map of the spatial branch features comprises:

[0016] compress the spatial branch features based on the CSAM module in the preset fusion model to obtain an aggregated map of each of the spatial branch features;

[0017] perform convolution processing on the aggregated map based on the separable convolution layer in the CSAM module to obtain a feature representation map that retains global information in the spatial branch features.

[0018] In an embodiment, the step of determining the spatial branch features based on the SFBM module in the preset fusion model to obtain frequency features comprises:

[0019] perform frequency domain branch processing on the spatial branch features based on the SFBM module in the preset fusion model to obtain a feature spectrum of the spatial branch features;

[0020] extract the amplitude component and the phase component of the corresponding spectrum of the spatial branch features from the feature spectrum based on the frequency bifurcation layer in the SFBM module;

[0021] Based on the point convolution layer in the SFBM module, the amplitude component and the phase component are used to determine the frequency dependence of the spatial branch feature, and a frequency feature is obtained.

[0022] In an embodiment, the step of determining the frequency dependence of the spatial branch feature based on the point convolution layer in the SFBM module, using the amplitude component and the phase component, includes:

[0023] Based on the point convolution layer in the SFBM module, the structure information of the stitched image is captured from the amplitude component, and the frequency information of the stitched image is captured from the phase component;

[0024] According to the structure information and the frequency information, the spectral band information of the stitched image is captured;

[0025] Based on the spectral band information, the frequency dependence of the spatial branch feature is determined to obtain a frequency feature.

[0026] In an embodiment, the step of performing frequency domain branch on the spatial branch feature based on the SFBM module in the preset fusion model to obtain a feature spectrum of the spatial branch feature includes:

[0027] Based on the frequency perceiver of the SFBM module in the preset fusion model, the spatial branch feature is switched to the frequency domain to obtain a feature frequency of each spatial branch feature;

[0028] Based on the decomposition linear layer in the frequency perceiver, the feature frequency is decomposed and linearly converted to obtain a nonlinear frequency representing nonlinear filtering;

[0029] Based on the frequency component layer in the frequency perceiver, the nonlinear frequency is processed by components to obtain a frequency component;

[0030] Based on the frequency component layer, the frequency component is reorganized to obtain a feature spectrum of the spatial branch feature.

[0031] In addition, to achieve the above-mentioned purpose, the present application also provides a multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network, which comprises:

[0032] An acquisition module is configured to acquire a multispectral image and a hyperspectral image of a remote sensing image.

[0033] The capturing module is configured to capture a spatial dependence relationship of spatial branch features of the multispectral image and the hyperspectral image based on a CSAM module in the preset fusion model, to obtain spatial features, wherein the spatial branch features are features obtained after spatial branch processing of the multispectral image and the hyperspectral image.

[0034] The determining module is configured to determine a frequency dependence of the spatial branch features based on an SFBM module in the preset fusion model, to obtain frequency features.

[0035] The fusion module is configured to fuse the spatial features and the frequency features based on a convolution layer in the preset fusion model, to obtain a fusion image of the multispectral image and the hyperspectral image.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the multispectral and hyperspectral image fusion method based on the spatial frequency collaborative network as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the steps of the multispectral and hyperspectral image fusion method based on the spatial frequency collaborative network as described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the multispectral and hyperspectral image fusion method based on the spatial frequency collaborative network as described above.

[0039] The one or more technical solutions provided by the present application have at least the following technical effects:

[0040] After obtaining the multispectral image and hyperspectral image of the remote sensing image, in order to extract more spatial features and spectral features from the multispectral image and hyperspectral image, so that the finally obtained image has higher spatial resolution, and then improve the clarity of the finally obtained image and its resolution ability to ground features, therefore, the multispectral image and hyperspectral image are fused by the preset fusion model, since the CSAM (Condensed Spatial Augmentation Module) module is used in the building process of the preset fusion model, therefore, the spatial dependence relationship of the spatial branch features of the multispectral image and hyperspectral image can be captured by using the CSAM module, and more detailed spatial features in the spatial branch features are obtained, and in order to extract more frequency features from the spatial branch features, therefore, the SFBM (Selective Frequency Bifurcated Module) module is set in the preset fusion model, so as to determine the frequency dependence of the spatial branch features by the SFBM module, and obtain the global frequency features of the spatial branch features, and then the spatial features and frequency features of the spatial branch features are fused by using the convolution layer in the preset fusion model, so that the finally obtained fusion image contains more frequency features and more detailed spatial features, the fusion image has more fine features, and then the quality of image fusion is improved. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0043] Figure 1 The flowchart provided by the first embodiment of the multispectral and hyperspectral image fusion method based on the spatial frequency collaborative network of the present application;

[0044] Figure 2 The simple framework diagram of the preset fusion model in the multispectral and hyperspectral image fusion method based on the spatial frequency collaborative network of the present application;

[0045] Figure 3 The flowchart provided by the second embodiment of the multispectral and hyperspectral image fusion method based on the spatial frequency collaborative network of the present application;

[0046] Figure 4A flowchart provided by the third embodiment of the multispectral and hyperspectral image fusion method based on the spatial frequency collaborative network of the present application;

[0047] Figure 5 A flowchart provided by the fourth embodiment of the multispectral and hyperspectral image fusion method based on the spatial frequency collaborative network of the present application;

[0048] Figure 6 A module structure diagram of the multispectral and hyperspectral image fusion device based on the spatial frequency collaborative network of the present application;

[0049] Figure 7 A device structure diagram of the hardware running environment involved in the multispectral and hyperspectral image fusion method based on the spatial frequency collaborative network of the present application.

[0050] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0052] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the drawings and specific embodiments of the specification.

[0053] The main solution of the present application is that the image fusion management platform acquires multispectral images and hyperspectral images of remote sensing images; based on the CSAM module in the preset fusion model, the spatial dependence relationship of the spatial branch features of the multispectral images and the hyperspectral images is captured to obtain spatial features, the spatial branch features are features after spatial branch processing of the multispectral images and the hyperspectral images; based on the SFBM module in the preset fusion model, the frequency dependence of the spatial branch features is determined to obtain frequency features; based on the convolution layer in the preset fusion model, the spatial features and the frequency features are fused to obtain the fusion image of the multispectral images and the hyperspectral images.

[0054] In the present embodiment, for the convenience of description, the following is described with the image fusion management platform as the execution subject.

[0055] Since the frequency domain features can provide a global view, capture the global correlation in the image, effectively supplement the local features in the spatial domain, and thus enhance the overall representation ability of the image, the current method captures the interaction between different modalities by using directional pair multi-head cross attention, thereby promoting the information transmission between modalities, and a SwinTransformer block is added after the cross attention to enhance the self-attention in the context to better fuse the multispectral image and the hyperspectral image. In addition, the self-attention in the vision is replaced by Fourier transform and a learnable filter, the amplitude and phase components of the image are exchanged in the Fourier frequency domain to enhance and adjust the frequency information, and the long space-time dependence is realized in the frequency domain. The image is decomposed into amplitude and phase components to effectively describe the structural information of the image. Although the Fourier transform has advantages in enhancing the frequency domain information capturing ability, it cannot effectively integrate the frequency domain and spatial features, thereby limiting the quality of image fusion.

[0056] The present application provides a solution, after obtaining the multispectral image and the hyperspectral image of the remote sensing image, in order to extract more spatial features and spectral features from the multispectral image and the hyperspectral image, so that the finally obtained image has higher spatial resolution, and thus the clarity and the resolution ability of the finally obtained image to the ground object features are improved. Therefore, the multispectral image and the hyperspectral image are fused by the preset fusion model. Since the CSAM module is used in the building process of the preset fusion model, the spatial dependence relationship of the spatial branch features of the multispectral image and the hyperspectral image can be captured by using the CSAM module, and more detailed spatial features in the spatial branch features can be obtained. In order to extract more frequency features from the spatial branch features, the SFBM module is set in the preset fusion model to determine the frequency dependence of the spatial branch features by the SFBM module, and the global frequency features of the spatial branch features are obtained. The spatial features and the frequency features of the spatial branch features are fused by using the convolution layer in the preset fusion model, so that the finally obtained fusion image contains more frequency features and more detailed spatial features, the fusion image has more fine features, and thus the quality of image fusion is improved.

[0057] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, an image fusion management platform, etc. capable of realizing the above functions. The present embodiment and the following embodiments will be described below taking the image fusion management platform as an example.

[0058] Based on this, the present application provides a multispectral and hyperspectral image fusion method based on a spatial frequency cooperative network, which is described with reference to Figure 1 , Figure 1A flowchart of a first embodiment of a multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network of the application.

[0059] In this embodiment, the multispectral and hyperspectral image fusion method based on the spatial frequency collaborative network comprises steps S10-S40:

[0060] Step S10, obtaining a multispectral image and a hyperspectral image of a remote sensing image;

[0061] It should be noted that the remote sensing image can be a visible image obtained by remotely sensing the electromagnetic wave information reflected or radiated by the target and recording it. The multispectral image can be a remote sensing image that can simultaneously obtain multiple optical spectral bands, usually containing several to tens of bands, and can provide more spectral information than the panchromatic image. The hyperspectral image can refer to a spectral image with a spectral resolution in the order of 10 ^ -2λ, which can simultaneously obtain two-dimensional spatial information and one-dimensional spectral information of the target, form a data cube, and thus provide more information than traditional multispectral images. Among them, the multispectral image is a high spatial resolution multispectral image; the hyperspectral image is a low spatial resolution hyperspectral image.

[0062] It can be understood that since the multispectral image has more spatial information and the hyperspectral image has more spectral information, and in subsequent fusion of the multispectral image and the hyperspectral image, the multispectral image and the hyperspectral image are needed as inputs, therefore, the multispectral image and the hyperspectral image of the remote sensing image need to be obtained.

[0063] Step S20, based on a CSAM module in a preset fusion model, capturing a spatial dependency relationship of spatial branch features of the multispectral image and the hyperspectral image, obtaining spatial features, the spatial branch features being features after spatial branch processing of the multispectral image and the hyperspectral image;

[0064] It should be noted that the CSAM module can balance the capture of extensive context information and detailed local details to obtain more effective and comprehensive spatial features in the spliced image. The spatial dependency relationship can be the mutual influence and correlation between adjacent pixels or regions, reflecting the spatial distribution law and internal relationship of the ground objects, which can help reveal the interaction and influence mechanism between the ground objects. The spatial features can be information and attributes related to the spatial position extracted from the spliced image.

[0065] It can be understood that, since the CSAM module can balance the capture of extensive context information and detailed local details, the spatial branch features of the spliced image are obtained after the spatial branch processing of the spliced image of the multispectral image and the hyperspectral image, the spatial dependence relationship of the spatial branch features is captured, more effective and comprehensive spatial features in the spatial branch features are obtained, in order to obtain more spatial features from the spatial branch features, therefore, the spatial dependence relationship of the spatial branch features is captured by the CSAM module, to obtain as many influences and correlations between features in the multispectral image and the hyperspectral image as possible, according to the influences and correlations, the spatial features of the spatial branch features are determined, to obtain more detailed spatial features, and more extensive and more detailed spatial features are provided for subsequent fusion.

[0066] In step S30, the SFBM module in the preset fusion model is used to determine the frequency dependence of the spatial branch features, to obtain frequency features.

[0067] It should be noted that the SFBM module can effectively retain the key frequency components of the spatial branch features and filter out redundant details. The frequency dependence can be the feature difference of electromagnetic waves of the same frequency (waveband) on a remote sensing image. Since the physical and chemical properties of features are different in different wavebands, the original features can be maximized. The frequency features can be the response characteristics of electromagnetic waves of different frequencies (wavebands) to ground objects, which are usually manifested as the differences in reflectivity, radiation intensity or scattering coefficient parameters in different wavebands.

[0068] It can be understood that, since the SFBM module can effectively retain the key frequency components of the spatial branch features and filter out redundant details, in order to obtain more detailed frequency domain features from the spatial branch features, the frequency dependence of the spatial branch features is determined by the SFBM module, to obtain remote sensing wavebands of different objects in the multispectral image and the hyperspectral image, to accurately distinguish the detailed information in the spatial branch features through these wavebands, and to obtain more detailed frequency features.

[0069] In step S40, the spatial features and the frequency features are fused by the convolution layer in the preset fusion model, to obtain a fusion image of the multispectral image and the hyperspectral image.

[0070] It can be understood that the frequency features and the spatial features are effectively integrated by the preset fusion model. The preset fusion model improves the accuracy and detail performance of the image in the fusion process, overcomes the technical difficulties of integrating frequency domain and spatial domain features in existing methods, and realizes higher quality image fusion.

[0071] In a specific implementation, with reference to Figure 2 The network uses a low-resolution hyperspectral image (LR-HSI) and high resolution multispectral image (HR-MSI) As input, generate high resolution hyperspectral image (HR-HSI) where r = H / h = W / w represents the up-sampling factor. First, the bicubic interpolated is stitched with Then, a 3x3 convolutional layer is applied to the stitched input to extract shallow features. The extracted shallow features are then fed into three spatial frequency residual groups (SFRGs) to further extract deep features.

[0072] To make full use of the information extracted by each layer of the deep neural network, the module introduces dense connections. These dense connections not only improve feature transmission, but also promote feature reuse and alleviate the gradient vanishing problem. Each SFRG integrates these dense connections to optimize the extraction of deep features. The SFRG also integrates the CSAM and SFBM modules, where the CSAM captures spatial dependencies through the compressed window multi-head self-attention mechanism (CW-MSA), and the SFBM focuses on selective frequency transformation to address frequency-based dependencies. These modules work together to enhance the model's ability to explore and capture complex relationships in multispectral and hyperspectral image fusion, which is crucial for achieving accurate image fusion.

[0073] The embodiment provides a multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network. After obtaining the multispectral image and the hyperspectral image of the remote sensing image, more spatial features and spectral features can be extracted from the multispectral image and the hyperspectral image, so that the finally obtained image has higher spatial resolution, and the clarity and the resolution capability of the finally obtained image to ground object features are improved. Therefore, the multispectral image and the hyperspectral image are fused by a preset fusion model. Since the CSAM module is used in the construction process of the preset fusion model, the spatial dependency of the spatial branch features of the multispectral image and the hyperspectral image can be captured by using the CSAM module, and more detailed spatial features in the spatial branch features are obtained. In order to extract more frequency features from the spatial branch features, the SFBM module is arranged in the preset fusion model, so as to determine the frequency dependency of the spatial branch features by using the SFBM module, and obtain the global frequency features of the spatial branch features. Then, the spatial features and the frequency features of the spatial branch features are fused by using the convolutional layer in the preset fusion model, so that the finally obtained fusion image contains more frequency features and more detailed spatial features, the fusion image has more fine features, and the quality of the image fusion is improved.

[0074] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and the subsequent will not be described. On this basis, please refer to Figure 3 , step S20 also includes steps S01-S03:

[0075] Step S01, based on the convolution layer in the preset fusion model, the spatial branch processing is performed on the splicing image of the multispectral image and the hyperspectral image, and the spatial branch feature of the splicing image is obtained;

[0076] Step S02, based on the CSAM module in the preset fusion model, the spatial branch feature is compressed to obtain the feature representation map of the spatial branch feature;

[0077] Step S03, based on the feature representation map, the self-attention mechanism in the CSAM module is executed, the spatial dependence relationship of the spatial branch feature is captured, and the spatial feature is obtained.

[0078] It should be noted that the spatial branch feature not only includes effective features, but also includes useless interference features. The feature representation map can be obtained by aggregating the original features through convolution to compress the original features into a vector map with rich features. The self-attention mechanism can be a neural network technology for enhancing sequence data processing, which can be a condensed window multi-head self-attention.

[0079] It can be understood that by compressing the spatial branch feature to obtain the feature representation map, the complexity of the calculation required for processing the feature is reduced, and the computing power required for running the preset fusion model is reduced, and the key global information in the spatial branch feature is retained. At the same time, the integrity of the feature is guaranteed, the calculation complexity of the fusion of the multispectral image and the hyperspectral image is reduced, the fusion efficiency is improved, and the global spatial feature after fusion is obtained according to the complete effective feature (feature representation map).

[0080] It can be understood that the condensed window multi-head self-attention (Condensed Window Multi-head Self-attention, CW-MSA) combines window-based global feature extraction with convolution-based local feature enhancement. By compressing the window feature through convolution, CW-MSA retains the key global information while reducing the calculation complexity. Effectively balance the needs of broad context understanding and meticulous local feature enhancement.

[0081] Further, step S02 can also include:

[0082] Based on the CSAM module in the preset fusion model, the spatial branch feature is compressed in spatial dimension to obtain an aggregation map of each spatial branch feature.

[0083] Convolve the aggregated graph based on the separable convolution layer in the CSAM module to obtain a feature representation graph that retains global information in the spatial branch feature.

[0084] It should be noted that the aggregated graph can show the correlation, similarity or difference between the spatial branch features in a graphical manner, so as to show the user in a graphical manner, so that the user can quickly understand the data structure, identify important features, and discover potential patterns or outliers.

[0085] It can be understood that, in order to reduce the complexity of the calculation when the features are fused, and to retain the key global information in the spatial branch feature, the spatial branch feature is first compressed and represented by a visual aggregated graph, so that the user can quickly understand the correlation, similarity and difference between the spatial branch features according to the aggregated graph, and make decisions according to these information.

[0086] It can be understood that, by convolving the aggregated graph, the number of features to be processed is reduced, thereby reducing the computational complexity of processing the features, further reducing the computing power required for processing the features, and thereby improving the rate of processing the features.

[0087] In a specific implementation, first, the original window feature is compressed into a representative feature map, which intuitively aggregates the information of the entire window, and the reference Figure 2 (C) Given input features It is first divided into HW / M 2 Local windows, each with a size of MxM. For a local window feature First, the spatial dimension is reduced from the original MxM to a smaller m x m through iterative deep convolution, generating a rough aggregated graph. Subsequently, the rough graph is refined using deep separable convolution to obtain a fine compressed representation While retaining the channel dimension to maintain the expressive ability of the attention map generated by each attention head. Then, cross-attention is performed using the query derived from the representative feature map and value The attention matrix is calculated based on the dot product interaction between the query and the key. The mathematical formula of the proposed CW-MSA is:

[0088] Q = X w W Q ,K c = X c W K ,V c = X w W V,

[0089]

[0090] where W Q ,W K ,W V is the projection matrix, denotes the aligned relative position embedding, which is obtained by interpolating the original embedding due to the difference in size of Q and K c . is a scalar used to adjust the result of the dot product. Meanwhile, h cross-attention functions are performed and the results are spliced together to realize multi-head cross-attention. In addition, a shift window division is applied on two consecutive CW-MSA.

[0091] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as the above embodiments can be referred to the above introduction, and will not be described in detail. On this basis, please refer to Figure 4 , step S30 further includes steps S31-S33:

[0092] Step S31, based on the SFBM module in the preset fusion model, performing frequency domain branch on the spatial branch feature to obtain a feature spectrum of the spatial branch feature;

[0093] Step S32, based on the frequency bifurcation layer in the SFBM module, extracting an amplitude component and a phase component of a corresponding spectrum of the spatial branch feature from the feature spectrum;

[0094] Step S33, based on the point convolution layer in the SFBM module, using the amplitude component and the phase component to determine the frequency dependence of the spatial branch feature to obtain a frequency feature.

[0095] It should be noted that the frequency domain branch can be divided according to the frequency range, because the reflection or radiation characteristics of different ground objects at different frequencies are different. The feature spectrum can be the spectrum of each spatial branch feature in the spliced image. The amplitude component can be the length of the vector composed of the real part and the imaginary part of a complex signal or waveform in the field of signal processing and communication. The phase component can describe the position of the wave or signal in space and time.

[0096] It can be understood that the spatial branch feature in the spliced image is subjected to frequency domain branch by the SFBM module to realize selective frequency bifurcation to capture the frequency-based dependence, that is, the SFBM can capture the complex frequency dependence often neglected in the spatial model. By differentiating the input features in different frequency domains, the diversified spectral information inherent in the multispectral and hyperspectral image fusion task can be processed more effectively.

[0097] It can be understood that the SFBM can effectively capture and integrate different information in the frequency domain, thereby improving the overall performance and accuracy of the multispectral and hyperspectral image fusion task.

[0098] Further, the step S33 further includes:

[0099] Based on the point convolution layer in the SFBM module, the structural information of the splicing image is captured from the amplitude component, and the frequency information of the splicing image is captured from the phase component;

[0100] According to the structural information and the frequency information, the spectral band information of the splicing image is captured;

[0101] Based on the spectral band information, the frequency-dependent determination of the spatial branch feature is performed to obtain a frequency feature.

[0102] It should be noted that the point convolution layer can be a layer that performs point-by-point multiplication operation between two sequences in the frequency domain. The structural information can be information about the shape, size and mutual relationship of the ground objects extracted from the remote sensing data. The frequency information can be the frequency of the remote sensing data about the shape, size and mutual relationship of the ground objects.

[0103] It can be understood that since directly applying 3x3 convolution to the amplitude component can cause spectral leakage and channel misplacement, the point convolution layer is set in the SFBM module to limit its operation to a single spatial position in the frequency domain through the point convolution layer, effectively prevent overlapping that can destroy the consistency of the channel structure, thereby maintaining the fidelity of the spectrum, further allowing accurate extraction and integration of spectral information between different frequency bands, minimizing the risk of artifacts and ensuring the integrity of the fused data.

[0104] On the contrary, the phase component encodes texture details and other fine information, so convolution is needed to effectively capture spatial information to ensure that the spectral band edge definition and texture consistency are accurately expressed.

[0105] It can be understood that the frequency domain feature is converted and analyzed by 2D-FFT, which not only improves the efficiency of feature processing, but also enhances the ability of the module to manage high-dimensional spectral information. By utilizing 2D-FFT, the SFBM can effectively capture and integrate complex spectral patterns in different frequency domains, thereby improving the overall performance and accuracy of the multispectral and hyperspectral image fusion task.

[0106] In a specific implementation, given a feature 2D-FFT is used to obtain the corresponding frequency representation:

[0107]

[0108] where F(·) denotes the two-dimensional fast Fourier transform (2D-FFT), and u and v represent the horizontal and vertical spatial frequencies in the Fourier spectrum X F , respectively. X F contains complex values, denoted as where and are the real and imaginary parts, respectively. Due to the conjugate symmetry, only half of the spatial dimensions are preserved after the FFT transformation, thus reducing the computational complexity of the network. Next, the amplitude and phase components are extracted from the spectrum, which are expressed as:

[0109]

[0110] where the amplitude component captures the structural information of the image, while the phase component encodes the high-frequency details, such as textures and subtle changes.

[0111] Based on the above-mentioned embodiments and the second embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as the above-mentioned embodiments can be referred to the above introduction, and the subsequent will not be described in detail. On this basis, please refer to Figure 5 , step S31 further includes steps S1-S4:

[0112] Step S1, based on the frequency sensor of the SFBM module in the preset fusion model, switching the spatial branch feature to the frequency domain to obtain the feature frequency of each spatial branch feature;

[0113] Step S2, based on the decomposition linear layer in the frequency sensor, decomposing and linearly converting the feature frequency to obtain a nonlinear frequency representing nonlinear filtering;

[0114] Step S3, based on the frequency component layer in the frequency sensor, performing component processing on the nonlinear frequency to obtain a frequency component;

[0115] Step S4, based on the frequency component layer, recombining the frequency component to obtain the feature spectrum of the spatial branch feature.

[0116] It can be understood that a filtering structure is constructed in the frequency domain through the frequency domain feedforward network, which effectively enhances or suppresses different frequency components, thereby better extracting and fusing the detail features in the frequency domain and the spatial domain. This method effectively improves the fidelity and quality of the fused image, ensuring that the complementary information from the hyperspectral image and the multispectral image is fully utilized.

[0117] In the implementation, since multiplication in the frequency domain is equivalent to circular convolution in the spatial domain, this property is utilized to construct a global filter by performing max pooling on the amplitude and average pooling on the phase. Subsequently, convolution operations are used to enhance the global filter and improve its effectiveness. Because multiplication in the frequency domain is equivalent to circular convolution in the spatial domain, multiplying this global filter with the processed features allows for selective preservation of key frequency information while suppressing irrelevant details. Finally, an inverse Fourier transform is used to convert the frequency domain features back to the image domain. The formula for this transformation is as follows:

[0118]

[0119] in, G(X) represents the feature components of amplitude and phase information after convolution integration. F ) represents a global filter. F -1 This represents the inverse Fourier transform, and the symbol ⊙ represents element-wise multiplication. Output X out This is a feature map of selective branching. The selective branching process effectively captures deep features and their interdependencies in the frequency domain, enhancing the overall image representation.

[0120] In practical implementation, the frequency domain feedforward network can be understood as a frequency multilayer perceptron, such as... Figure 2 As shown in (d), this network constructs a filtering structure in the frequency domain by combining a grouped linear layer with the GELU activation function. The initial step of FreqMLP is to apply an FFT to the input features, transforming them into the frequency domain. The transformed signal then passes through a grouped linear layer, which performs grouping and linear transformations similar to frequency decomposition. Next, a nonlinearity is introduced through GELU activation, acting as a nonlinear filter. A second grouped linear layer further refines the frequency components, improving the representation by recombination and fine-tuning them. This architecture resembles a frequency domain filter, selectively enhancing or suppressing different frequency components. Therefore, it can effectively manipulate the signal's frequency band, extract subtle features, and improve the overall representation. By operating in the frequency domain, FreqMLP can leverage the inherent characteristics of Fourier transform signals, providing a high-level mechanism for processing complex frequency information.

[0121] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the multispectral and hyperspectral image fusion method based on spatial frequency cooperative networks in this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0122] This application also provides a multispectral and hyperspectral image fusion device based on a spatial frequency cooperative network. Please refer to [link / reference]. Figure 6 The multispectral and hyperspectral image fusion device based on a spatial frequency cooperative network includes:

[0123] The acquisition module 10 is configured to acquire a multispectral image and a hyperspectral image of a remote sensing image.

[0124] The capture module 20 is configured to capture spatial dependence of spatial branch features of the multispectral image and the hyperspectral image based on a CSAM module in the preset fusion model, to obtain spatial features, the spatial branch features being features after spatial branch processing of the multispectral image and the hyperspectral image.

[0125] The determination module 30 is configured to determine frequency dependence of the spatial branch features based on an SFBM module in the preset fusion model, to obtain frequency features.

[0126] The fusion module 40 is configured to fuse the spatial features and the frequency features based on a convolution layer in the preset fusion model, to obtain a fusion image of the multispectral image and the hyperspectral image.

[0127] Optionally, the capture module 20 is further configured to perform spatial branch processing on a spliced image of the multispectral image and the hyperspectral image based on a convolution layer in the preset fusion model, to obtain spatial branch features of the spliced image; perform spatial dependence capture on the spatial branch features based on a self-attention mechanism in the CSAM module by using the feature representation map of the spatial branch features, to obtain spatial features.

[0128] Optionally, the capture module 20 is further configured to perform spatial dimension compression on the spatial branch features based on the CSAM module in the preset fusion model, to obtain an aggregation map of each of the spatial branch features; perform convolution processing on the aggregation map based on a separable convolution layer in the CSAM module, to obtain a feature representation map that retains global information in the spatial branch features.

[0129] Optionally, the determination module 30 is further configured to perform frequency branch processing on the spatial branch features based on the SFBM module in the preset fusion model, to obtain a feature spectrum of the spatial branch features; extract an amplitude component and a phase component of a corresponding spectrum of the spatial branch features from the feature spectrum based on a frequency bifurcation layer in the SFBM module; and perform frequency dependence determination on the spatial branch features based on a point convolution layer in the SFBM module by using the amplitude component and the phase component, to obtain frequency features.

[0130] Optionally, the determining module 30 is further configured to capture structural information of the stitched image from the amplitude component and frequency information of the stitched image from the phase component based on a point convolution layer in the SFBM module; capture spectral band information of the stitched image according to the structural information and the frequency information; and perform frequency-dependent determination on the spatial branch feature based on the spectral band information to obtain a frequency feature.

[0131] Optionally, the determining module 30 is further configured to switch the spatial branch feature to a frequency domain based on a frequency perceiver of the SFBM module in the preset fusion model to obtain a feature frequency of each of the spatial branch features; perform decomposition and linear conversion on the feature frequency based on a decomposition linear layer in the frequency perceiver to obtain a nonlinear frequency representing nonlinear filtering; perform component processing on the nonlinear frequency based on a frequency component layer in the frequency perceiver to obtain a frequency component; and recombine the frequency component based on the frequency component layer to obtain a feature spectrum of the spatial branch feature.

[0132] The multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network provided in the application adopts the multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network in the above embodiment, and can solve the technical problem that the frequency domain and spatial features cannot be effectively integrated, thereby limiting the quality of image fusion. Compared with the prior art, the multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network provided in the application has the same beneficial effects as the multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network provided in the above embodiment, and other technical features of the multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0133] The application provides a multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network, which comprises at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network in the above embodiment one.

[0134] The following refers to Figure 7This document illustrates a schematic diagram of a multispectral and hyperspectral image fusion device based on a spatial frequency cooperative network, suitable for implementing embodiments of this application. The multispectral and hyperspectral image fusion device based on a spatial frequency cooperative network in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The multispectral and hyperspectral image fusion device based on spatial frequency cooperative networks shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0135] like Figure 7 As shown, the multispectral and hyperspectral image fusion device based on a spatial frequency cooperative network may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the multispectral and hyperspectral image fusion device based on the spatial frequency cooperative network. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the multispectral and hyperspectral image fusion device based on a spatial frequency cooperative network to wirelessly or wiredly communicate with other devices to exchange data. Although the figure shows a multispectral and hyperspectral image fusion device based on a spatial frequency cooperative network with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0136] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0137] The multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network provided by the present application adopts the multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network in the above-mentioned embodiments, which can solve the technical problem that the frequency domain and spatial features cannot be effectively integrated, thereby limiting the quality of image fusion. Compared with the prior art, the multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network provided by the present application has the same beneficial effects as the multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network provided by the above-mentioned embodiments, and other technical features in the multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0138] It should be understood that various parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0139] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0140] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network in the above-mentioned embodiments.

[0141] The computer readable storage medium provided in the application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to electrical wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0142] The above computer readable storage medium can be included in a spatial frequency cooperative network-based multispectral and hyperspectral image fusion device, or can exist independently without being assembled into the spatial frequency cooperative network-based multispectral and hyperspectral image fusion device.

[0143] The above computer readable storage medium carries one or more programs, which, when executed by the spatial frequency cooperative network-based multispectral and hyperspectral image fusion device, cause the spatial frequency cooperative network-based multispectral and hyperspectral image fusion device to: acquire a multispectral image and a hyperspectral image of a remote sensing image; based on a CSAM module in a preset fusion model, capture a spatial dependence relationship of spatial branch features of the multispectral image and the hyperspectral image, to obtain spatial features, the spatial branch features being features after spatial branch processing of the multispectral image and the hyperspectral image; based on an SFBM module in the preset fusion model, determine frequency dependence of the spatial branch features, to obtain frequency features; based on a convolution layer in the preset fusion model, fuse the spatial features and the frequency features, to obtain a fusion image of the multispectral image and the hyperspectral image.

[0144] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0145] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0146] The modules involved in the embodiments of the present application can be implemented in software or hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0147] The readable storage medium provided by the application is a computer readable storage medium, and the computer readable storage medium stores computer readable program instructions (namely, a computer program) for executing the above-mentioned multi-spectral and hyper-spectral image fusion method based on a spatial frequency cooperative network, and can solve the technical problem that frequency domain and spatial features cannot be effectively integrated, thereby limiting the quality of image fusion. Compared with the prior art, the computer readable storage medium provided by the application has the same beneficial effects as the multi-spectral and hyper-spectral image fusion method based on a spatial frequency cooperative network provided by the above-mentioned embodiments, and details are not repeated here.

[0148] The application further provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the multi-spectral and hyper-spectral image fusion method based on a spatial frequency cooperative network as described above.

[0149] The computer program product provided by the application can solve the technical problem that frequency domain and spatial features cannot be effectively integrated, thereby limiting the quality of image fusion. Compared with the prior art, the computer program product provided by the application has the same beneficial effects as the multi-spectral and hyper-spectral image fusion method based on a spatial frequency cooperative network provided by the above-mentioned embodiments, and details are not repeated here.

[0150] The above-mentioned is only part of the embodiments of the application, and does not limit the patent scope of the application, and any equivalent structural transformation made by using the content of the specification and drawings of the application, or direct / indirect application in other related technical fields is included in the patent protection scope of the application.

Claims

1. A method for multispectral and hyperspectral image fusion based on a spatial frequency cooperative network, characterized in that, The method includes: Acquire multispectral and hyperspectral images from remote sensing data; Based on the convolutional layer in the preset fusion model, spatial branching processing is performed on the stitched image of the multispectral image and the hyperspectral image to obtain the spatial branching features of the stitched image. The spatial branching features are the features after spatial branching processing of the multispectral image and the hyperspectral image. Based on the CSAM module in the preset fusion model, the spatial branch features are compressed to obtain the feature representation map of the spatial branch features, wherein the CSAM module is a compressed spatial enhancement module; Based on the feature representation map, the self-attention mechanism in the CSAM module is executed to capture the spatial dependencies of the spatial branch features and obtain spatial features; Based on the SFBM module in the preset fusion model, the spatial branch feature is frequency domain branched to obtain the feature spectrum of the spatial branch feature. The SFBM module is a selective frequency bifurcation module. Based on the frequency bifurcation layer in the SFBM module, the amplitude and phase components of the spectrum corresponding to the spatial branch feature are extracted from the feature spectrum; Based on the point convolutional layer in the SFBM module, the frequency dependence of the spatial branch features is determined by using the amplitude component and the phase component, thereby obtaining the frequency features; Based on the convolutional layer in the preset fusion model, the spatial features and the frequency features are fused to obtain a fused image of the multispectral image and the hyperspectral image.

2. The method as described in claim 1, characterized in that, The step of compressing the spatial branch features to obtain the feature representation map of the spatial branch features based on the CSAM module in the preset fusion model includes: Based on the CSAM module in the preset fusion model, the spatial dimension of the spatial branch features is compressed to obtain an aggregated graph of each spatial branch feature. The aggregated graph is convolved using the separable convolutional layer in the CSAM module to obtain a feature representation map that retains global information in the spatial branch features.

3. The method as described in claim 1, characterized in that, The step of determining the frequency features by using the amplitude component and the phase component to determine the frequency dependence of the spatial branch features based on the point convolutional layer in the SFBM module includes: Based on the point convolutional layer in the SFBM module, the structural information of the stitched image is captured from the amplitude component, and the frequency information of the stitched image is captured from the phase component; The spectral band information of the stitched image is captured based on the structural information and the frequency information; Based on the spectral band information, the frequency dependence of the spatial branching features is determined to obtain frequency features.

4. The method as described in claim 1, characterized in that, The step of performing frequency domain branching on the spatial branch features based on the SFBM module in the preset fusion model to obtain the feature spectrum of the spatial branch features includes: Based on the frequency sensor of the SFBM module in the preset fusion model, the spatial branch features are switched to the frequency domain to obtain the feature frequencies of each spatial branch feature. Based on the decomposition linear layer in the frequency sensor, the characteristic frequency is decomposed and linearly transformed to obtain the nonlinear frequency characterizing the nonlinear filter. Based on the frequency component layer in the frequency sensing, the nonlinear frequency is processed to obtain frequency components. The frequency components are recombined based on the frequency component layer to obtain the feature spectrum of the spatial branch feature.

5. A multispectral and hyperspectral image fusion device based on a spatial frequency cooperative network, characterized in that, The device includes: The acquisition module is used to acquire multispectral and hyperspectral images of remote sensing images; A capture module is used to perform spatial branching processing on the stitched image of the multispectral image and the hyperspectral image based on the convolutional layer in a preset fusion model to obtain the spatial branching features of the stitched image. The spatial branching features are the features after spatial branching processing of the multispectral image and the hyperspectral image. Based on the CSAM module in the preset fusion model, the spatial branching features are compressed to obtain a feature representation map of the spatial branching features. The CSAM module is a spatial compression enhancement module. Based on the feature representation map, the self-attention mechanism in the CSAM module is executed to capture the spatial dependencies of the spatial branching features to obtain spatial features. The determination module is used to perform frequency domain branching on the spatial branch features based on the SFBM module in the preset fusion model to obtain the feature spectrum of the spatial branch features, wherein the SFBM module is a selective frequency bifurcation module; based on the frequency bifurcation layer in the SFBM module, the amplitude component and phase component of the spectrum corresponding to the spatial branch features are extracted from the feature spectrum; based on the point convolution layer in the SFBM module, the frequency dependence of the spatial branch features is determined using the amplitude component and the phase component to obtain frequency features; The fusion module is used to fuse the spatial features and the frequency features based on the convolutional layer in the preset fusion model to obtain a fused image of the multispectral image and the hyperspectral image.

6. A multispectral and hyperspectral image fusion device based on a spatial frequency cooperative network, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multispectral and hyperspectral image fusion method based on a spatial frequency cooperative network as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the multispectral and hyperspectral image fusion method based on a spatial frequency cooperative network as described in any one of claims 1 to 4.

8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the multispectral and hyperspectral image fusion method based on a spatial frequency cooperative network as described in any one of claims 1 to 4.

Images