Multispectral and hyperspectral image fusion method based on spatial frequency collaborative network

Abstract

The invention discloses a multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network, relates to the technical field of remote sensing, and discloses the multispectral and hyperspectral image fusion method based on the spatial frequency collaborative network, comprising: acquiring a multispectral image and a hyperspectral image of a remote sensing image; on the basis of a CSAM module in a preset fusion model, capturing a spatial dependency relationship of the spatial branch features of the multispectral image and the hyperspectral image to obtain spatial features; based on an SFBM module in a preset fusion model, performing frequency dependence determination on the spatial branch features to obtain frequency features; and based on a convolutional layer in a preset fusion model, fusing the spatial features and the frequency features to obtain a fused image of the multispectral image and the hyperspectral image. By integrating the spatial domain and frequency domain features, the model can capture and fuse complex information in multispectral and hyperspectral images more accurately, and the image fusion quality is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of remote sensing technology, and in particular to a multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network. Background Art

[0002] Hyperspectral images can provide hundreds to thousands of narrow-band spectral information to capture the unique characteristics of various materials, while multispectral imaging systems provide high spatial resolution. In order to generate hyperspectral images with high spatial resolution, the fusion research of multispectral and hyperspectral images combines the advantages of both.

[0003] Since frequency domain features can provide a global view, capture global correlations in images, and effectively supplement local features in the spatial domain, thereby enhancing the overall representation ability of images, directional paired multi-head cross attention is currently used to capture the interaction between different modalities, thereby promoting information transfer between modalities. A SwinTransformer block is added after cross attention to enhance self-attention in the context to better fuse multispectral images and hyperspectral images. Fourier transform and learnable filters can also replace self-attention in vision, exchange the amplitude and phase components of the image in the Fourier frequency domain to enhance and adjust the frequency information, realize long-term spatiotemporal dependencies in the frequency domain, and decompose the image into amplitude and phase components to effectively describe the structural information of the image. Although Fourier transform has advantages in enhancing the ability to capture frequency domain information, it cannot effectively integrate frequency domain and spatial features, thereby limiting the quality of image fusion.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a multispectral and hyperspectral image fusion method based on a space-frequency collaborative network, aiming to solve the technical problem that the frequency domain and spatial features cannot be effectively integrated, thereby limiting the quality of image fusion.

[0006] To achieve the above objectives, the present application proposes a multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network, the method comprising:

[0007] Acquire multispectral and hyperspectral images of remote sensing images;

[0008] Based on the CSAM module in the preset fusion model, the spatial dependency relationship between the spatial branch features of the multispectral image and the hyperspectral image is captured to obtain spatial features, where the spatial branch features are features after the multispectral image and the hyperspectral image are processed by spatial branching;

[0009] Based on the SFBM module in the preset fusion model, determining the frequency dependency of the spatial branch feature to obtain a frequency feature;

[0010] Based on the convolution layer in the preset fusion model, the spatial feature and the frequency feature are fused to obtain a fused image of the multispectral image and the hyperspectral image.

[0011] In one embodiment, the step of capturing the spatial dependency relationship between the spatial branch features of the multispectral image and the hyperspectral image based on the CSAM module in the preset fusion model, and obtaining the spatial features includes:

[0012] Based on the convolution layer in the preset fusion model, a spatial branching process is performed on the spliced ​​image of the multispectral image and the hyperspectral image to obtain a spatial branching feature of the spliced ​​image;

[0013] Based on the CSAM module in the preset fusion model, compressing the spatial branch features to obtain a feature representation graph of the spatial branch features;

[0014] Based on the feature representation graph, the self-attention mechanism in the CSAM module is executed to capture the spatial dependency of the spatial branch features and obtain spatial features.

[0015] In one embodiment, the step of compressing the spatial branch features to obtain a feature representation graph of the spatial branch features based on the CSAM module in the preset fusion model includes:

[0016] Based on the CSAM module in the preset fusion model, compress the spatial dimension of the spatial branch features to obtain an aggregation graph of each of the spatial branch features;

[0017] The aggregate graph is convolved based on the separable convolutional layer in the CSAM module to obtain a feature representation graph that retains the global information in the spatial branch features.

[0018] In one embodiment, the step of determining the frequency dependency of the spatial branch feature based on the SFBM module in the preset fusion model to obtain the frequency feature includes:

[0019] Based on the SFBM module in the preset fusion model, the spatial branch feature is branched in the frequency domain to obtain a characteristic spectrum of the spatial branch feature;

[0020] Based on the frequency bifurcation layer in the SFBM module, extracting the amplitude component and the phase component of the spectrum corresponding to the spatial branch feature from the characteristic spectrum;

[0021] Based on the point convolution layer in the SFBM module, the frequency dependency of the spatial branch feature is determined by utilizing the amplitude component and the phase component to obtain the frequency feature.

[0022] In one embodiment, the step of determining the frequency dependency of the spatial branch feature based on the point convolution layer in the SFBM module using the amplitude component and the phase component to obtain the frequency feature includes:

[0023] Based on the point convolution layer in the SFBM module, capturing the structural information of the stitched image from the amplitude component and capturing the frequency information of the stitched image from the phase component;

[0024] capturing spectral band information of the stitched image according to the structural information and the frequency information;

[0025] Based on the spectral band information, the frequency dependency of the spatial branch feature is determined to obtain a frequency feature.

[0026] In one embodiment, the step of performing frequency domain branching on the spatial branch feature based on the SFBM module in the preset fusion model to obtain a characteristic spectrum of the spatial branch feature includes:

[0027] Based on the frequency sensor of the SFBM module in the preset fusion model, the spatial branch features are switched to the frequency domain to obtain the characteristic frequency of each of the spatial branch features;

[0028] Based on the decomposition linear layer in the frequency sensor, the characteristic frequency is decomposed and linearly transformed to obtain a nonlinear frequency representing nonlinear filtering;

[0029] Based on the frequency component layer in the frequency perception, component processing is performed on the nonlinear frequency to obtain frequency components;

[0030] The frequency components are reorganized based on the frequency component layer to obtain a characteristic spectrum of the spatial branch feature.

[0031] In addition, to achieve the above purpose, the present application also proposes a multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network, and the multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network includes:

[0032] An acquisition module, used to acquire multispectral images and hyperspectral images of remote sensing images;

[0033] A capture module, for capturing the spatial dependency relationship between the spatial branch features of the multispectral image and the hyperspectral image based on the CSAM module in the preset fusion model, and obtaining spatial features, wherein the spatial branch features are features after the multispectral image and the hyperspectral image are processed by spatial branching;

[0034] A determination module, configured to determine the frequency dependency of the spatial branch feature based on the SFBM module in the preset fusion model to obtain a frequency feature;

[0035] A fusion module is used to fuse the spatial features with the frequency features based on the convolution layer in the preset fusion model to obtain a fused image of the multispectral image and the hyperspectral image.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a multispectral and hyperspectral image fusion device based on a spatial-frequency collaborative network, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multispectral and hyperspectral image fusion method based on a spatial-frequency collaborative network as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multispectral and hyperspectral image fusion method based on the spatial-frequency collaborative network as described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the multispectral and hyperspectral image fusion method based on spatial-frequency collaborative network as described above.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] After obtaining the multispectral image and hyperspectral image of the remote sensing image, in order to extract more spatial features and spectral features from the multispectral image and the hyperspectral image, so that the final image has a higher spatial resolution, thereby improving the clarity of the final image and its ability to distinguish the features of the ground objects, the multispectral image and the hyperspectral image are fused through a preset fusion model. Since the preset fusion model uses the CSAM (Condensed Spatial Augmentation Module) module during the construction process, the CSAM module can be used to capture the spatial dependency of the spatial branch features of the multispectral image and the hyperspectral image, and obtain more detailed spatial features in the spatial branch features. In order to extract more frequency features from the spatial branch features, the SFBM (Selective Frequency Bifurcated Module) is set in the preset fusion model. A selective frequency bifurcation module (SFBM) is used to determine the frequency dependence of spatial branch features through the SFBM module, obtain the global frequency characteristics of spatial branch features, and then use the convolution layer in the preset fusion model to fuse the spatial features and frequency characteristics of the spatial branch features, so that the final fused image contains more frequency features and more detailed spatial features, and the fused image has more refined features, thereby improving the quality of image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the multispectral and hyperspectral image fusion method based on a spatial-frequency collaborative network of the present application;

[0044] Figure 2 This is a simplified framework diagram of a preset fusion model in the multispectral and hyperspectral image fusion method based on a spatial-frequency collaborative network in this application;

[0045] Figure 3 A flowchart diagram of a second embodiment of a multispectral and hyperspectral image fusion method based on a spatial-frequency collaborative network of the present application;

[0046] Figure 4A flowchart diagram of the third embodiment of the multispectral and hyperspectral image fusion method based on a spatial-frequency collaborative network of the present application;

[0047] Figure 5 A flowchart diagram of a fourth embodiment of the multispectral and hyperspectral image fusion method based on a spatial-frequency collaborative network of the present application is provided;

[0048] Figure 6 This is a schematic diagram of the module structure of a multi-spectral and hyper-spectral image fusion device based on a spatial-frequency collaborative network according to an embodiment of the present application;

[0049] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the multispectral and hyperspectral image fusion method based on the spatial frequency cooperative network in the embodiment of the present application.

[0050] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the embodiment of the present application is: the image fusion management platform obtains the multispectral image and the hyperspectral image of the remote sensing image; based on the CSAM module in the preset fusion model, the spatial dependency relationship of the spatial branch features of the multispectral image and the hyperspectral image is captured to obtain the spatial feature, and the spatial branch feature is the feature after the multispectral image and the hyperspectral image are processed by spatial branching; based on the SFBM module in the preset fusion model, the frequency dependency of the spatial branch feature is determined to obtain the frequency feature; based on the convolution layer in the preset fusion model, the spatial feature and the frequency feature are fused to obtain the fused image of the multispectral image and the hyperspectral image.

[0054] In this embodiment, for ease of description, the following description is made with the image fusion management platform as the execution entity.

[0055] Since frequency domain features can provide a global view, capture global correlations in images, and effectively supplement local features in the spatial domain, thereby enhancing the overall representation ability of images, directional paired multi-head cross attention is currently used to capture the interaction between different modalities, thereby promoting information transfer between modalities. A SwinTransformer block is added after cross attention to enhance self-attention in the context to better fuse multispectral images and hyperspectral images. Fourier transform and learnable filters can also replace self-attention in vision, exchange the amplitude and phase components of the image in the Fourier frequency domain to enhance and adjust the frequency information, realize long-term spatiotemporal dependencies in the frequency domain, and decompose the image into amplitude and phase components to effectively describe the structural information of the image. Although Fourier transform has advantages in enhancing the ability to capture frequency domain information, it cannot effectively integrate frequency domain and spatial features, thereby limiting the quality of image fusion.

[0056] The present application provides a solution. After acquiring the multispectral image and the hyperspectral image of the remote sensing image, in order to extract more spatial features and spectral features from the multispectral image and the hyperspectral image, so that the final image has a higher spatial resolution, thereby improving the clarity of the final image and its ability to resolve the features of the ground objects, the multispectral image and the hyperspectral image are fused through a preset fusion model. Since the preset fusion model adopts a CSAM module during the construction process, the CSAM module can be used to capture the spatial dependency relationship of the spatial branch features of the multispectral image and the hyperspectral image, and obtain more detailed spatial features in the spatial branch features. In order to extract more frequency features from the spatial branch features, an SFBM module is set in the preset fusion model to determine the frequency dependency of the spatial branch features through the SFBM module, obtain the global frequency features of the spatial branch features, and then use the convolution layer in the preset fusion model to fuse the spatial features and frequency features of the spatial branch features, so that the final fused image contains more frequency features and more detailed spatial features, improve the fused image has more refined features, and thus improve the quality of image fusion.

[0057] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an image fusion management platform, etc. The following takes the image fusion management platform as an example to illustrate this embodiment and the following embodiments.

[0058] Based on this, the embodiment of the present application provides a multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network, referring to Figure 1 , Figure 1This is a flowchart of the first embodiment of the multispectral and hyperspectral image fusion method based on spatial-frequency collaborative network of the present application.

[0059] In this embodiment, the multispectral and hyperspectral image fusion method based on spatial frequency collaborative network includes steps S10 to S40:

[0060] Step S10, acquiring a multispectral image and a hyperspectral image of a remote sensing image;

[0061] It should be noted that remote sensing images can be visual images obtained by sensing the electromagnetic wave information reflected or radiated by the target from a distance and recording it. Multispectral images can be remote sensing images that can simultaneously obtain multiple optical spectrum bands, usually containing several to a dozen bands, and can provide more spectral information than full-color images. Hyperspectral images can refer to images with a spectral resolution of 10 ^ Spectral images within the range of -2λ can simultaneously obtain the two-dimensional spatial information and one-dimensional spectral information of the target, forming a data cube, thereby providing richer information than traditional multispectral images. Among them, multispectral images are multispectral images with high spatial resolution; hyperspectral images are hyperspectral images with low spatial resolution.

[0062] It is understandable that since the multispectral image has more spatial information and the hyperspectral image has more spectral information, and when the multispectral image and the hyperspectral image are subsequently fused, the multispectral image and the hyperspectral image are needed as input. Therefore, it is necessary to obtain the multispectral image and the hyperspectral image of the remote sensing image.

[0063] Step S20, based on the CSAM module in the preset fusion model, the spatial dependency relationship of the spatial branch features of the multispectral image and the hyperspectral image is captured to obtain spatial features, where the spatial branch features are features after the multispectral image and the hyperspectral image are processed by spatial branching;

[0064] It should be noted that the CSAM module can capture extensive contextual information and fine local details in a balanced manner to obtain more effective and comprehensive spatial features in the stitched image. Spatial dependency can refer to the mutual influence and association between adjacent pixels or regions, reflecting the spatial distribution law and internal connection of objects, and can help reveal the interaction and influence mechanism between objects. Spatial features can be information and attributes related to spatial position extracted from the stitched image.

[0065] It can be understood that since the CSAM module can capture extensive contextual information and fine local details in a balanced manner, the spatial branch features of the stitched image of the multispectral image and the hyperspectral image are obtained after spatial branching processing. By capturing the spatial dependencies of the spatial branch features, more effective and comprehensive spatial features in the spatial branch features are obtained. In order to obtain more spatial features from the spatial branch features, the spatial dependencies of the spatial branch features are captured by the CSAM module to obtain as many influences and associations as possible between the objects in the multispectral image and the hyperspectral image. According to these influences and associations, the spatial features of the spatial branch features are determined to obtain more detailed spatial features, providing a wider range and more refined spatial features for subsequent fusion.

[0066] Step S30, based on the SFBM module in the preset fusion model, determining the frequency dependency of the spatial branch feature to obtain a frequency feature;

[0067] It should be noted that the SFBM module can effectively retain the key frequency components of the spatial branch characteristics and filter out unnecessary details. Frequency dependence can be the characteristic differences of electromagnetic waves of the same frequency (band) on remote sensing images. Due to the physical and chemical properties of ground objects in different bands, the original characteristics can be retained to the maximum extent. Frequency characteristics can be the response characteristics of electromagnetic waves of different frequencies (bands) to surface objects, which are usually manifested as differences in parameters such as reflectivity, radiation intensity or scattering coefficient in different bands.

[0068] It can be understood that since the SFBM module can effectively retain the key frequency components of the spatial branch features and filter out redundant details, in order to obtain more refined frequency domain features from the spatial branch features, the frequency dependence of the spatial branch features is determined through the SFBM module to obtain the remote sensing bands of different objects in the multispectral images and hyperspectral images, and these bands are used to accurately distinguish the detail information in the spatial branch features to obtain more refined frequency features.

[0069] Step S40: Based on the convolution layer in the preset fusion model, the spatial feature and the frequency feature are fused to obtain a fused image of the multispectral image and the hyperspectral image.

[0070] It can be understood that the frequency features and spatial features are effectively integrated through the preset fusion model. The preset fusion model improves the accuracy and detail performance of the image during the fusion process, overcomes the technical difficulties of integrating frequency domain and spatial domain features in the existing methods, and achieves higher quality image fusion.

[0071] In the specific implementation, refer to Figure 2 , the network is based on low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral imaging (HR-MSI) As input, a high-resolution hyperspectral image (HR-HSI) is generated Where r = H / h = W / w represents the upsampling multiple. First, the bicubic interpolation and A 3×3 convolutional layer is then applied to the concatenated input to extract shallow features. The extracted shallow features are then fed into three spatial frequency residual groups (SFRGs) to further extract deep features.

[0072] In order to make full use of the information extracted from each layer of the deep neural network, the module introduces dense connections. These dense connections not only improve feature transfer, but also promote feature reuse and alleviate the gradient vanishing problem. Each SFRG integrates these dense connections to optimize the extraction of deep features. SFRG also integrates CSAM and SFBM modules, where CSAM captures spatial dependencies through the compressed window multi-head self-attention mechanism (CW-MSA), while SFBM focuses on selective frequency transformation to address frequency-based dependencies. These modules work together to enhance the model's ability to explore and capture complex relationships in multispectral and hyperspectral image fusion, which is crucial to achieving accurate image fusion.

[0073] The present embodiment provides a multispectral and hyperspectral image fusion method based on a spatial frequency collaborative network. After obtaining the multispectral image and the hyperspectral image of the remote sensing image, in order to extract more spatial features and spectral features from the multispectral image and the hyperspectral image, so that the final image has a higher spatial resolution, thereby improving the clarity of the final image and its ability to distinguish ground features, the multispectral image and the hyperspectral image are fused through a preset fusion model. Since the preset fusion model adopts a CSAM module during the construction process, the CSAM module can be used to capture the spatial dependency relationship of the spatial branch features of the multispectral image and the hyperspectral image, so as to obtain more detailed spatial features in the spatial branch features. In order to extract more frequency features from the spatial branch features, an SFBM module is set in the preset fusion model to determine the frequency dependency of the spatial branch features through the SFBM module, obtain the global frequency features of the spatial branch features, and then use the convolution layer in the preset fusion model to fuse the spatial features and frequency features of the spatial branch features, so that the final fused image contains more frequency features and more detailed spatial features, so as to improve the fused image to have more refined features, thereby improving the quality of image fusion.

[0074] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , step S20 also includes steps S01 to S03:

[0075] Step S01, based on the convolution layer in the preset fusion model, performing spatial branching processing on the spliced ​​image of the multispectral image and the hyperspectral image to obtain the spatial branching features of the spliced ​​image;

[0076] Step S02, based on the CSAM module in the preset fusion model, compressing the spatial branch features to obtain a feature representation graph of the spatial branch features;

[0077] Step S03, based on the feature representation graph, execute the self-attention mechanism in the CSAM module to capture the spatial dependency of the spatial branch features and obtain spatial features.

[0078] It should be noted that the spatial branch features include not only effective features but also useless interference features. The feature representation graph may be a graph that aggregates the original features through convolution to compress the original features into a vector graph with rich features. The self-attention mechanism may be a neural network technology for enhancing sequence data processing, specifically, a compressed window multi-head self-attention.

[0079] It can be understood that by compressing the spatial branch features to obtain a feature representation map, the complexity of the calculations required to process the features is reduced, and the computing power required to run the preset fusion into the model is reduced, and the key global information in the spatial branch features is retained. While ensuring the integrity of the features, the computational complexity of fusing multispectral images with hyperspectral images is reduced, the fusion efficiency is improved, and the fused spatial features are made global based on the complete and effective features (feature representation map).

[0080] It can be understood that the window-based global feature extraction is combined with the local feature enhancement of convolution through Condensed Window Multi-head Self-attention (CW-MSA). By convolutionally compressing the window features, CW-MSA retains key global information while reducing the computational complexity. It effectively balances the needs of broad context understanding and detailed local feature enhancement.

[0081] Furthermore, step S02 may also include:

[0082] Based on the CSAM module in the preset fusion model, compress the spatial dimension of the spatial branch features to obtain an aggregation graph of each of the spatial branch features;

[0083] The aggregate graph is convolved based on the separable convolutional layer in the CSAM module to obtain a feature representation graph that retains the global information in the spatial branch features.

[0084] It should be noted that the aggregation graph can graphically display the correlation, similarity or difference between the features of each spatial branch, so as to display it to the user in a graphical way, allowing the user to quickly understand the data structure, identify important features, and discover potential patterns or outliers.

[0085] It is understandable that in order to reduce the complexity of calculation during feature fusion and retain the key global information in the spatial branch features, the spatial branch features are first compressed and presented through a visual aggregation graph. Then, the user can quickly understand the correlation, similarity and difference between the spatial branch features according to the aggregation graph, and use this information.

[0086] It can be understood that by performing convolution processing on the aggregate graph, the number of features that need to be processed is reduced, thereby reducing the computational complexity of processing the features, and further reducing the computing power required for processing the features, thereby increasing the rate at which the features are processed.

[0087] In the specific implementation, the original window features are first compressed into a representative feature map, which intuitively aggregates the information of the entire window. Figure 2 (C) Given input features It is first divided into HW / M 2 local windows, each with a size of M×M. For a local window feature First, the spatial dimension is reduced from the original M×M to a smaller m×m through iterative depthwise convolution to generate a coarse aggregation graph. Subsequently, the coarse graph is refined using depthwise separable convolution to obtain a finely compressed representation. At the same time, the channel dimension is preserved to maintain the expressiveness of the attention map generated by each attention head. Then, the query generated from the original window features is used With the key derived from the representative feature map Sum Perform cross attention. The attention matrix is ​​calculated based on the dot product interaction between the query and the key. The mathematical formula of the proposed CW-MSA is:

[0088] Q=X w W Q ,K c =X c W K ,V c =X w W V,

[0089]

[0090] Among them, W Q ,W K ,W V is the projection matrix, Indicates the relative position embedding of the alignment. Since the size of Q is similar to K c , so it is obtained by interpolating the original embedding. is a scalar used to adjust the result of the dot product. The criss-cross attention function is performed h times at the same time, and the results are concatenated together to achieve multi-head criss-cross attention. In addition, a shift window partition is applied on two consecutive CW-MSAs.

[0091] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 4 , step S30 also includes steps S31 to S33:

[0092] Step S31, based on the SFBM module in the preset fusion model, performing frequency domain branching on the spatial branch feature to obtain a characteristic spectrum of the spatial branch feature;

[0093] Step S32, based on the frequency bifurcation layer in the SFBM module, extracting the amplitude component and the phase component of the spectrum corresponding to the spatial branch feature from the characteristic spectrum;

[0094] Step S33, based on the point convolution layer in the SFBM module, using the amplitude component and the phase component, the frequency dependency of the spatial branch feature is determined to obtain the frequency feature.

[0095] It should be noted that the frequency domain branches can be divided according to the frequency range, because different objects have different reflection or radiation characteristics at different frequencies. The characteristic spectrum can be the spectrum of each spatial branch feature in the spliced ​​image. The amplitude component can be the length of the vector composed of the real and imaginary parts of a complex signal or waveform in the field of signal processing and communication. The phase component can describe the position of the fluctuation or signal in space and time.

[0096] It can be understood that the spatial branch features in the spliced ​​image are branched in the frequency domain through the SFBM module to achieve selective frequency bifurcation to capture frequency-based dependencies, that is, because SFBM can capture complex frequency dependencies that are often ignored in spatial models. By differentiating the input features in different frequency domains, the diverse spectral information inherent in multispectral and hyperspectral image fusion tasks can be more effectively processed.

[0097] It can be understood that SFBM can effectively capture and integrate different information in the frequency domain, thereby improving the overall performance and accuracy of multispectral and hyperspectral image fusion tasks.

[0098] Furthermore, step S33 also includes:

[0099] Based on the point convolution layer in the SFBM module, capturing the structural information of the stitched image from the amplitude component and capturing the frequency information of the stitched image from the phase component;

[0100] capturing spectral band information of the stitched image according to the structural information and the frequency information;

[0101] Based on the spectral band information, the frequency dependency of the spatial branch feature is determined to obtain a frequency feature.

[0102] It should be noted that the point convolution layer can be a layer that performs point-by-point product operations between two sequences in the frequency domain. The structural information can be information about the shape, size, and relationship of surface objects extracted from remote sensing data. The frequency information can be the frequency of the remote sensing data about the shape, size, and relationship of surface objects.

[0103] It is understandable that since directly applying 3×3 convolution to the amplitude component may lead to spectral leakage and channel misalignment, a point convolution layer is set in the SFBM module to limit its operation to a single spatial position in the frequency domain through the point convolution layer, and effectively prevent overlap that may destroy the consistency of the channel structure, thereby maintaining the fidelity of the spectrum. Further, it allows accurate extraction and integration of spectral information between different frequency bands, minimizing the risk of artifacts and ensuring the integrity of the fused data.

[0104] In contrast, the phase component encodes textural details and other fine information, so convolution is needed to effectively capture spatial information to ensure that edge sharpness and texture consistency between spectral bands are accurately represented.

[0105] It can be understood that the frequency domain features are converted and analyzed through two-dimensional fast Fourier transform (2D-FFT), which not only improves the efficiency of feature processing, but also enhances the module's ability to manage high-dimensional spectral information. By utilizing 2D-FFT, SFBM can effectively capture and integrate complex spectral patterns in different frequency domains, thereby improving the overall performance and accuracy of multispectral and hyperspectral image fusion tasks.

[0106] In the specific implementation, given a feature Use 2D-FFT to obtain the corresponding frequency representation:

[0107]

[0108] Where F(·) represents the two-dimensional fast Fourier transform (2D-FFT), u and v represent the Fourier spectrum X F The horizontal and vertical spatial frequencies in X F Contains complex values, represented by in and are the real part and the imaginary part respectively. Due to the conjugate symmetry, the features after FFT conversion only retain half of the spatial dimension, thus reducing the computational complexity of the network. Next, the amplitude and phase components are extracted from the spectrum, and their expressions are:

[0109]

[0110] Among them, the amplitude component The structural information of the image is captured, while the phase component It encodes high-frequency details such as texture and subtle changes.

[0111] Based on the above embodiment and the second embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the above embodiment can refer to the above introduction, and will not be repeated in the following. Figure 5 , step S31 also includes steps S1 to S4:

[0112] Step S1, based on the frequency sensor of the SFBM module in the preset fusion model, switching the spatial branch features to the frequency domain to obtain the characteristic frequency of each of the spatial branch features;

[0113] Step S2, based on the decomposition linear layer in the frequency sensor, decomposing and linearly transforming the characteristic frequency to obtain a nonlinear frequency representing nonlinear filtering;

[0114] Step S3, performing component processing on the nonlinear frequency based on the frequency component layer in the frequency perception to obtain frequency components;

[0115] Step S4: reorganize the frequency components based on the frequency component layer to obtain a characteristic spectrum of the spatial branch feature.

[0116] It can be understood that a filter structure is constructed in the frequency domain through the frequency domain feedforward network, which effectively enhances or suppresses different frequency components, thereby better extracting and fusing the detail features in the frequency domain and the spatial domain. This method effectively improves the fidelity and quality of the fused image and ensures that the complementary information from the hyperspectral image and the multispectral image is fully utilized.

[0117] In the specific implementation, since multiplication in the frequency domain is equivalent to circular convolution in the spatial domain, this property is used to construct a global filter by performing maximum pooling on the amplitude and average pooling on the phase. Subsequently, the global filter is enhanced by a convolution operation to improve its effectiveness. Since multiplication in the frequency domain is equivalent to circular convolution in the spatial domain, by multiplying this global filter with the processed features, key frequency information can be selectively retained while suppressing irrelevant details. Finally, the frequency domain features are converted back to the image domain by inverse Fourier transform. The formula for the above conversion is as follows:

[0118]

[0119] in, Represents the characteristic components of the amplitude and phase information after convolution integration, G(X F ) represents the global filter. F -1 represents the inverse Fourier transform, and the symbol ⊙ represents element-wise multiplication. Output X out is the feature map of the selective branch. The selective branching process effectively captures the deep features in the frequency domain and their interdependencies, enhancing the overall image representation.

[0120] In the specific implementation, the frequency domain feedforward network can be understood as a frequency multilayer perceptron, such as Figure 2 As shown in (d), the network constructs a filtering structure in the frequency domain by combining a group linear layer with a GELU activation function. The initial step of FreqMLP is to apply FFT to the input features to transform them into the frequency domain. The converted signal then passes through a group linear layer, which performs grouping and linear transformation similar to frequency decomposition. Next, nonlinearity is introduced through GELU activation, which plays a role of nonlinear filtering. The second group linear layer further refines the frequency components and improves the representation effect by recombining and fine-tuning them. This architecture is similar to a frequency domain filter that can selectively enhance or suppress different frequency components. Therefore, it can effectively manipulate the frequency band of the signal, extract subtle features and improve the overall representation. By operating in the frequency domain, FreqMLP is able to exploit the inherent characteristics of Fourier transform signals and provide an advanced mechanism for processing complex frequency information.

[0121] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the multispectral and hyperspectral image fusion method based on the spatial-frequency collaborative network of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0122] This application also provides a multi-spectral and hyperspectral image fusion device based on a spatial frequency collaborative network, please refer to Figure 6 , the multispectral and hyperspectral image fusion device based on spatial frequency collaborative network includes:

[0123] An acquisition module 10 is used to acquire multispectral images and hyperspectral images of remote sensing images;

[0124] A capture module 20 is used to capture the spatial dependency relationship between the spatial branch features of the multispectral image and the hyperspectral image based on the CSAM module in the preset fusion model to obtain spatial features, wherein the spatial branch features are features after the multispectral image and the hyperspectral image are processed by spatial branching;

[0125] A determination module 30, configured to determine the frequency dependency of the spatial branch feature based on the SFBM module in the preset fusion model to obtain a frequency feature;

[0126] The fusion module 40 is used to fuse the spatial feature with the frequency feature based on the convolution layer in the preset fusion model to obtain a fused image of the multispectral image and the hyperspectral image.

[0127] Optionally, the capture module 20 is further used to perform spatial branch processing on the stitched image of the multispectral image and the hyperspectral image based on the convolution layer in the preset fusion model to obtain spatial branch features of the stitched image; based on the CSAM module in the preset fusion model, compress the spatial branch features to obtain a feature representation graph of the spatial branch features; based on the feature representation graph, execute the self-attention mechanism in the CSAM module to capture the spatial dependencies of the spatial branch features to obtain spatial features.

[0128] Optionally, the capture module 20 is also used to compress the spatial branch features in spatial dimensions based on the CSAM module in the preset fusion model to obtain an aggregate graph of each of the spatial branch features; and to convolve the aggregate graph based on the separable convolution layer in the CSAM module to obtain a feature representation graph that retains the global information in the spatial branch features.

[0129] Optionally, the determination module 30 is also used to perform frequency domain branching on the spatial branch feature based on the SFBM module in the preset fusion model to obtain a characteristic spectrum of the spatial branch feature; based on the frequency bifurcation layer in the SFBM module, extract the amplitude component and phase component of the spectrum corresponding to the spatial branch feature from the characteristic spectrum; based on the point convolution layer in the SFBM module, use the amplitude component and the phase component to determine the frequency dependence of the spatial branch feature to obtain a frequency feature.

[0130] Optionally, the determination module 30 is also used to capture the structural information of the stitched image from the amplitude component and the frequency information of the stitched image from the phase component based on the point convolution layer in the SFBM module; capture the spectral band information of the stitched image according to the structural information and the frequency information; and determine the frequency dependence of the spatial branch feature based on the spectral band information to obtain the frequency feature.

[0131] Optionally, the determination module 30 is also used to switch the spatial branch features to the frequency domain based on the frequency sensor of the SFBM module in the preset fusion model to obtain the characteristic frequency of each of the spatial branch features; based on the decomposition linear layer in the frequency sensor, decompose and linearly transform the characteristic frequencies to obtain the nonlinear frequencies that characterize the nonlinear filtering; based on the frequency component layer in the frequency perception, perform component processing on the nonlinear frequencies to obtain frequency components; based on the frequency component layer, reorganize the frequency components to obtain the characteristic spectrum of the spatial branch features.

[0132] The multispectral and hyperspectral image fusion device based on spatial frequency collaborative network provided by the present application adopts the multispectral and hyperspectral image fusion method based on spatial frequency collaborative network in the above-mentioned embodiment, which can solve the technical problem that the frequency domain and spatial features cannot be effectively integrated, thereby limiting the quality of image fusion. Compared with the prior art, the beneficial effects of the multispectral and hyperspectral image fusion device based on spatial frequency collaborative network provided by the present application are the same as the beneficial effects of the multispectral and hyperspectral image fusion method based on spatial frequency collaborative network provided by the above-mentioned embodiment, and the other technical features in the multispectral and hyperspectral image fusion device based on spatial frequency collaborative network are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0133] The present application provides a multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network. The multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multispectral and hyperspectral image fusion method based on the spatial frequency collaborative network in the above-mentioned embodiment one.

[0134] Reference below Figure 7, which shows a schematic diagram of the structure of a multispectral and hyperspectral image fusion device based on a spatial frequency cooperative network suitable for implementing the embodiment of the present application. The multispectral and hyperspectral image fusion device based on a spatial frequency cooperative network in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions: tablet computers), PMPs (Portable Media Players: portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The multi-spectral and hyper-spectral image fusion device based on the spatial-frequency cooperative network shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0135] like Figure 7 As shown, the multispectral and hyperspectral image fusion device based on the spatial frequency cooperative network may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: ReadOnly Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: RandomAccess Memory) 1004. In the RAM 1004, various programs and data required for the operation of the multispectral and hyperspectral image fusion device based on the spatial frequency cooperative network are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the multispectral and hyperspectral image fusion device based on the spatial frequency cooperative network to communicate with other devices wirelessly or by wire to exchange data. Although the multispectral and hyperspectral image fusion device based on the spatial frequency cooperative network with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0136] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0137] The multispectral and hyperspectral image fusion device based on spatial frequency collaborative network provided by the present application adopts the multispectral and hyperspectral image fusion method based on spatial frequency collaborative network in the above-mentioned embodiment, which can solve the technical problem that the frequency domain and spatial features cannot be effectively integrated, thereby limiting the quality of image fusion. Compared with the prior art, the beneficial effects of the multispectral and hyperspectral image fusion device based on spatial frequency collaborative network provided by the present application are the same as the beneficial effects of the multispectral and hyperspectral image fusion method based on spatial frequency collaborative network provided by the above-mentioned embodiment, and other technical features in the multispectral and hyperspectral image fusion device based on spatial frequency collaborative network are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0138] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0140] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the multispectral and hyperspectral image fusion method based on a spatial-frequency collaborative network in the above-mentioned embodiment.

[0141] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0142] The computer-readable storage medium may be included in the multispectral and hyperspectral image fusion device based on the spatial-frequency collaborative network; or may exist independently without being assembled into the multispectral and hyperspectral image fusion device based on the spatial-frequency collaborative network.

[0143] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a multispectral and hyperspectral image fusion device based on a spatial-frequency collaborative network, the multispectral and hyperspectral image fusion device based on a spatial-frequency collaborative network: obtains a multispectral image and a hyperspectral image of a remote sensing image; based on the CSAM module in a preset fusion model, captures the spatial dependency of the spatial branch features of the multispectral image and the hyperspectral image to obtain spatial features, where the spatial branch features are features after the multispectral image and the hyperspectral image are subjected to spatial branching processing; based on the SFBM module in the preset fusion model, determines the frequency dependency of the spatial branch features to obtain frequency features; based on the convolutional layer in the preset fusion model, fuses the spatial features with the frequency features to obtain a fused image of the multispectral image and the hyperspectral image.

[0144] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0147] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the multispectral and hyperspectral image fusion method based on the spatial frequency cooperative network, and can solve the technical problem that the frequency domain and spatial features cannot be effectively integrated, thereby limiting the quality of image fusion. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the multispectral and hyperspectral image fusion method based on the spatial frequency cooperative network provided by the above embodiment, and will not be repeated here.

[0148] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multispectral and hyperspectral image fusion method based on a spatial-frequency collaborative network as described above.

[0149] The computer program product provided by the present application can solve the technical problem that the frequency domain and spatial features cannot be effectively integrated, thereby limiting the quality of image fusion. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the multispectral and hyperspectral image fusion method based on the spatial frequency cooperative network provided by the above embodiment, and will not be repeated here.

[0150] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A multispectral and hyperspectral image fusion method based on spatial frequency collaborative network, characterized in that: The method includes: Acquire multispectral and hyperspectral images of remote sensing images; Based on the CSAM module in the preset fusion model, the spatial dependency relationship between the spatial branch features of the multispectral image and the hyperspectral image is captured to obtain spatial features, where the spatial branch features are features after the multispectral image and the hyperspectral image are processed by spatial branching; Based on the SFBM module in the preset fusion model, determining the frequency dependency of the spatial branch feature to obtain a frequency feature; Based on the convolution layer in the preset fusion model, the spatial feature and the frequency feature are fused to obtain a fused image of the multispectral image and the hyperspectral image.

2. The method according to claim 1, characterized in that The step of capturing the spatial dependency relationship between the spatial branch features of the multispectral image and the hyperspectral image based on the CSAM module in the preset fusion model to obtain the spatial features includes: Based on the convolution layer in the preset fusion model, a spatial branching process is performed on the spliced ​​image of the multispectral image and the hyperspectral image to obtain a spatial branching feature of the spliced ​​image; Based on the CSAM module in the preset fusion model, compressing the spatial branch features to obtain a feature representation graph of the spatial branch features; Based on the feature representation graph, the self-attention mechanism in the CSAM module is executed to capture the spatial dependency of the spatial branch features and obtain spatial features.

3. The method according to claim 2, characterized in that The step of compressing the spatial branch features to obtain a feature representation graph of the spatial branch features based on the CSAM module in the preset fusion model comprises: Based on the CSAM module in the preset fusion model, compress the spatial dimension of the spatial branch features to obtain an aggregation graph of each of the spatial branch features; The aggregate graph is convolved based on the separable convolutional layer in the CSAM module to obtain a feature representation graph that retains the global information in the spatial branch features.

4. The method according to claim 1, characterized in that The step of determining the frequency dependency of the spatial branch feature based on the SFBM module in the preset fusion model to obtain the frequency feature includes: Based on the SFBM module in the preset fusion model, the spatial branch feature is branched in the frequency domain to obtain a characteristic spectrum of the spatial branch feature; Based on the frequency bifurcation layer in the SFBM module, extracting the amplitude component and the phase component of the spectrum corresponding to the spatial branch feature from the characteristic spectrum; Based on the point convolution layer in the SFBM module, the frequency dependency of the spatial branch feature is determined by utilizing the amplitude component and the phase component to obtain the frequency feature.

5. The method according to claim 4, characterized in that The step of determining the frequency dependency of the spatial branch feature based on the point convolution layer in the SFBM module by using the amplitude component and the phase component to obtain the frequency feature comprises: Based on the point convolution layer in the SFBM module, capturing the structural information of the stitched image from the amplitude component and capturing the frequency information of the stitched image from the phase component; capturing spectral band information of the stitched image according to the structural information and the frequency information; Based on the spectral band information, the frequency dependency of the spatial branch feature is determined to obtain a frequency feature.

6. The method according to claim 4, characterized in that The step of performing frequency domain branching on the spatial branch feature based on the SFBM module in the preset fusion model to obtain a characteristic spectrum of the spatial branch feature comprises: Based on the frequency sensor of the SFBM module in the preset fusion model, the spatial branch features are switched to the frequency domain to obtain the characteristic frequency of each of the spatial branch features; Based on the decomposition linear layer in the frequency sensor, the characteristic frequency is decomposed and linearly transformed to obtain a nonlinear frequency representing nonlinear filtering; Based on the frequency component layer in the frequency perception, component processing is performed on the nonlinear frequency to obtain frequency components; The frequency components are reorganized based on the frequency component layer to obtain a characteristic spectrum of the spatial branch feature.

7. A multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network, characterized in that: The device comprises: An acquisition module, used to acquire multispectral images and hyperspectral images of remote sensing images; A capture module, for capturing the spatial dependency relationship between the spatial branch features of the multispectral image and the hyperspectral image based on the CSAM module in the preset fusion model, and obtaining spatial features, wherein the spatial branch features are features after the multispectral image and the hyperspectral image are processed by spatial branching; A determination module, configured to determine the frequency dependency of the spatial branch feature based on the SFBM module in the preset fusion model to obtain a frequency feature; A fusion module is used to fuse the spatial features with the frequency features based on the convolution layer in the preset fusion model to obtain a fused image of the multispectral image and the hyperspectral image.

8. A multispectral and hyperspectral image fusion device based on a spatial frequency collaborative network, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multispectral and hyperspectral image fusion method based on a spatial-frequency collaborative network as described in any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multispectral and hyperspectral image fusion method based on a spatial-frequency collaborative network are implemented as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the steps of the multispectral and hyperspectral image fusion method based on a spatial-frequency cooperative network as claimed in any one of claims 1 to 6.

Images