A spectral image fusion method based on neural operator and progressive resampling

Abstract

The application belongs to the technical field of image processing, and discloses a spectral image fusion method based on a neural operator and progressive resampling. Three different scale copies are extracted from a low-resolution hyperspectral image and a high-resolution multispectral image respectively; feature extraction is respectively performed on the different scale copies, the obtained features are connected in a spectral dimension to serve as an initial spectral image function; the initial spectral image function is input into a spectral fusion neural operator module to obtain a function u; the obtained function u is added to a low-resolution hyperspectral image after up-sampling to obtain a high-resolution hyperspectral target image. The application obtains an image with high resolution in both the spectrum and the space, and greatly reduces the calculation complexity.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a spectral image fusion method based on neural operators and progressive resampling. Background Art

[0002] Hyperspectral imaging technology utilizes hyperspectral sensors to simultaneously scan a target area using continuously segmented bands, producing hyperspectral images that combine spatial and spectral information. Due to their rich spatial and spectral signatures, hyperspectral images play a key role in many fields, including transformation detection, medical diagnosis, and biodiversity conservation. However, due to hardware limitations of imaging platforms, obtaining an image with both spatial and spectral high resolution is extremely difficult. Therefore, spectral image fusion technology, as an alternative approach, aims to extract the rich spatial and spectral signatures contained in low-resolution hyperspectral images and high-resolution multispectral images, respectively, and fuse them together to produce an image with both spectral and spatial high resolution. Therefore, how to efficiently extract the spectral and spatial information from spectral images and fuse them has become a pressing issue.

[0003] In recent years, with the advancement of technology, deep learning-based solutions have significantly outperformed traditional techniques in nearly all vision fields. In the field of spectral image fusion, a large number of methods based on convolutional neural networks and Transformers have emerged, all of which have achieved considerable success. However, existing convolutional neural network architectures suffer from an inherent bias: these models are limited by the locality of the convolution operation, gathering information only from the pixel domain and lacking the ability to explicitly capture global dependencies. Transformers, on the other hand, lack an inductive bias for images and are limited in capturing fine-grained features, leading to poor performance on small datasets. Furthermore, models based on these two methods do not transfer well between image datasets of varying grid sizes. This is because they learn instance-to-instance mappings between finite-dimensional spaces, rather than infinite-dimensional spaces, during training. Therefore, when faced with new spectral image datasets, the models need to be retrained, resulting in unnecessary waste of resources. Summary of the Invention

[0004] The purpose of the present invention is to provide a spectral image fusion method based on neural operators and progressive resampling to obtain images with high resolution in both spectral and spatial aspects while greatly reducing the computational complexity.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A spectral image fusion method based on a neural operator and progressive resampling, the spectral image fusion method based on a neural operator and progressive resampling comprising:

[0007] Step S1, obtaining a low-resolution hyperspectral image LrHSI and a high-resolution multispectral image HrMSI of the same target;

[0008] Step S2, performing bilinear interpolation upsampling on the low-resolution hyperspectral image LrHSI to obtain an upsampled hyperspectral image Up-LrHSI having the same spatial size as the high-resolution multispectral image HrMSI, inputting the upsampled hyperspectral image Up-LrHSI and the high-resolution multispectral image HrMSI into a multimodal multi-scale module, and the multimodal multi-scale module outputs three copies L1, L2, L3 and H1, H2, H3 of different scales corresponding to the upsampled hyperspectral image Up-LrHSI and the high-resolution multispectral image HrMSI, respectively;

[0009] Step S3: extract features from the replicas L1, L2, L3 and H1, H2, H3 of different scales respectively, and connect the obtained features in the spectral dimension as the initial spectral image function a;

[0010] Step S4: input the initial spectral image function a into the spectral fusion neural operator module, which performs the following operations: performing a spectral linear transformation on the initial spectral image function a to obtain a function v0, performing three kernel integrations on the function v0 step by step to obtain functions v1, v2, and v3, aggregating the four functions v0, v1, v2, and v3 using a deep supervision module and performing a spectral inverse linear transformation to obtain a function u; wherein each kernel integration includes a progressive resampling integral and a Galigin integral;

[0011] Step S5: Add the obtained function u to the upsampled hyperspectral image Up-LrHSI in step S2 to obtain a high-resolution hyperspectral target image HrHSI.

[0012] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.

[0013] Preferably, in step S2, the multimodal multi-scale module outputs three copies L1, L2, L3 and H1, H2, H3 of different scales corresponding to the upsampled hyperspectral image Up-LrHSI and the high-resolution multispectral image HrMSI, respectively, including:

[0014] Let the sizes of the upsampled hyperspectral image Up-LrHSI and the high-resolution multispectral image HrMSI be H×W×S and H×W×s, respectively, where H and W are the height and width of the spectral image, S and s are the number of spectra of the upsampled hyperspectral image Up-LrHSI and the high-resolution multispectral image HrMSI, respectively;

[0015] The upsampled hyperspectral image Up-LrHSI is taken as the replica L1, with a size of H×W×S and a grouping number of S. Then the first replication is performed and the replication result is grouped in the spectral dimension to obtain a replica H2 with a size of in is the number of groups, 2 is the number of spectra in each group; copy L1 is copied for the second time, and the copied results are grouped on the spectral dimension to obtain copy L3, the size of which is in is the number of groups, 4 is the number of spectra in each group; the high-resolution multispectral image HrMSI is used as a copy H1 with a size of H×W×s, and then the first spatial double downsampling is performed to obtain a copy H2 with a size of Then, based on the copy H2, a second spatial double downsampling is performed to obtain the copy H3, which has a size of

[0016] Preferably, in step S3, feature extraction is performed on the replicas L1, L2, L3 and H1, H2, H3 of different scales respectively, and the obtained features are connected in the spectral dimension as the initial spectral image function a, including:

[0017] Perform group convolution on the replica L1 and the replicas L2 and L3 grouped by the spectral dimension to extract spectral features, and connect the extracted spectral features on the spectral dimension to obtain feature L ′ ;

[0018] Perform 3×3 convolution with a kernel size of 3 on copy H1 to extract spatial features, and perform 3×3 convolution with a kernel size of 3 on copies H2 and H3 to extract spatial features, and then perform spatial upsampling to the same size as copy H1; use the three spatial features obtained by channel attention processing, that is, adjust the weight ratio of each spatial feature in the channel dimension, and connect the three spatial features after adjusting the weight ratio in the spectral dimension to obtain feature H ′ ;

[0019] The feature L ′ and feature H ′ Connect in the spectral dimension to obtain the initial spectral image function a.

[0020] Preferably, in step S4, the discrete form of the progressive resampling integral is defined as follows:

[0021]

[0022] The definition of the Galigin integral is as follows:

[0023]

[0024] Among them, Ω PR is the subspace obtained by resampling the original integral space Ω, v(x) is the value of the original function at the query point x, v(y) is the value of the original function at the key point y, K(v(x), v(y)) is the integral kernel function, which is used to measure the degree of correlation between v(x) and v(y), which is reflected as a weight in the formula, v(y i ) is the discretization representation of v(y), which is determined by the specific discretization degree of the integral space, and l refers to the subspace Ω PR The starting point in , and l∈[1,n v -m v ],m v is space Ω PR The number of all discrete points, l+m v Refers to the subspace Ω PR The end point in n v Refers to the end point in the original integration space Ω.

[0025] Preferably, in step S5, the obtained function u is added to the upsampled hyperspectral image Up-LrHSI in step S2 by point-by-point addition.

[0026] This paper presents a spectral image fusion method based on neural operators and progressive resampling, revealing a new spectral image fusion model that achieves high interpretability by learning the functional relationship between LrHSI, HrMSI, and HrHSI in infinite-dimensional space. By leveraging the discrete invariance of neural operators, the challenge of generalization to arbitrary scales can be naturally avoided. By combining the Galilkin-type integral with a newly proposed progressive resampling scheme, the computational complexity of the model is reduced while maintaining performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is an architecture diagram of the spectral image fusion method based on neural operators and progressive resampling of the present invention;

[0028] Figure 2 This is a structural diagram of the multi-modal and multi-level module of the present invention;

[0029] Figure 3 Flowchart of the kernel integration operation of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0032] In one of the embodiments, a spectral image fusion method based on neural operators and progressive resampling is provided. The method adopts a neural operator architecture to learn spectral image functions rather than mappings between instances. It has excellent generalization effects on different grids and introduces a progressive resampling method to reduce kernel integral related points, thereby reducing the number of parameters and computational complexity of the model.

[0033] Specifically, such as Figure 1 As shown, the spectral image fusion method based on neural operators and progressive resampling in this embodiment includes the following steps:

[0034] Step S1: Acquire a low-resolution hyperspectral image LrHSI and a high-resolution multispectral image HrMSI of the same target.

[0035] The acquired low-resolution hyperspectral image (LrHSI) has a low resolution, and object details and edges may not be clear because each pixel captures limited information. At the same time, low-resolution hyperspectral images (LrHSI) capture a relatively wide spectral range, typically consisting of dozens or even hundreds of discrete bands. Hyperspectral images provide detailed spectral information, enabling more precise identification and analysis of different substances.

[0036] The higher resolution of high-resolution multispectral images (HrMSI) means that each pixel in the image represents a smaller area. Object details and edges are typically clearer in high-resolution images because each pixel captures more detail. At the same time, high-resolution multispectral images (HrMSI) capture fewer spectral bands than hyperspectral images, typically containing less spectral information within a few bands.

[0037] Step S2: The low-resolution hyperspectral image LrHSI is upsampled by bilinear interpolation to obtain an upsampled hyperspectral image Up-LrHSI with the same spatial size as the high-resolution multispectral image HrMSI. The upsampled hyperspectral image Up-LrHSI and the high-resolution multispectral image HrMSI are input into the multi-modal multi-magnitude module (M5) for feature extraction. Three copies of different scales, L1, L2, L3 and H1, H2, H3, are obtained from the upsampled hyperspectral image Up-LrHSI and the high-resolution multispectral image HrMSI, respectively.

[0038] like Figure 2 As shown in Figure 2, the process of obtaining three copies of different scales L1, L2, L3 and H1, H2, H3 from the upsampled hyperspectral image Up-LrHSI and the high-resolution multispectral image HrMSI is as follows:

[0039] A. Let the sizes of the upsampled hyperspectral image Up-LrHSI and the high-resolution multispectral image HrMSI be H×W×S and H×W×s, respectively, where H and W are the height and width of the spectral image, S and s are the number of spectra of the upsampled hyperspectral image Up-LrHSI and the high-resolution multispectral image HrMSI, respectively.

[0040] B. Take the upsampled hyperspectral image Up-LrHSI as the copy L1, the size is H×W×S, the number of groups is S, and then perform the first replication, and group the replication results in the spectral dimension to obtain the copy L2, the size is in is the number of groups, 2 is the number of spectra in each group; copy L1 is copied for the second time, and the copied results are grouped on the spectral dimension to obtain copy L3, the size of which is in is the number of groups, and 4 is the number of spectra in each group. The high-resolution multispectral image HrMSI is taken as a copy H1 with a size of H×W×s, and then the first spatial double downsampling is performed to obtain a copy H2 with a size of Then perform a second downsampling in the space to obtain a copy H3 of size

[0041] Extracting features at different scales yields features at different scales, including semantic and detailed features. This improves the model's ability to recognize objects or scenes of varying scales and enhances its generalization capabilities. Combining features from different modalities yields more representative features, thereby enhancing the model's accuracy and robustness.

[0042] Step S3: Extract features from copies L1, L2, L3 and H1, H2, H3 of different scales respectively, and connect the obtained features in the spectral dimension as the initial spectral image function a, as shown in Figure 2 The specific operations are as follows:

[0043] Step S31: Perform group convolution (GC) on the replica L1 and the replicas L2 and L3 grouped on the spectral dimension to extract spectral features, and stack the obtained spectral features on the spectral dimension to obtain feature L ′ , size is H×W×3S.

[0044] Step S32: Perform a 3×3 convolution with a kernel size of 3 on copy H1 to extract spatial features, and perform a 3×3 convolution with a kernel size of 3 on copies H2 and H3 to extract spatial features, and then upsample them to the same size as the copy H1; stack the obtained features in the spectral dimension, and use channel self-attention (CA) to redefine the entire feature map, adjust the weight ratio of each feature map in the channel dimension, and obtain feature H ′ , size is H×W×s.

[0045] Step S33: Set the feature L ′ and feature H ′ The initial spectral image function a is obtained by stacking in the spectral dimension, with a size of H×W×(3S+s).

[0046] Step S4: Input the initial spectral image function a into the spectral fusion neural operator module (Spatial-Spectral Fusion Neural Operator, SFNO), and the spectral fusion neural operator module performs the following operations: perform spectral linear transformation on the initial spectral image function a Get function v0. Perform three kernel integrations on function v0 step by step to get functions v1, v2 and v3. Use the deep supervision module (concatenate) to aggregate the four functions v0, v1, v2 and v3 and perform spectral linear inverse transformation. Get the function u.

[0047] Step S41: Perform spectral linear transformation on the initial spectral image function a The resulting function v0 is of size H × W × lifting_dim, where lifting_dim is the potential spatial dimension of the function during kernel integration. Spectral linear transformation expands the spectral dimension of the image function, enhancing the feature learning capability of subsequent kernel integration.

[0048] Step S42: Perform three kernel integrations on the function v0 step by step to obtain functions v1, v2 and v3, as shown in Figure 3Each kernel integral shown includes a progressive resampling integral and a Galigin integral. For the function to be processed, a progressive resampling integral (PR) is performed first, followed by a Galigin integral (GI) to obtain the output function. The discrete form of the progressive resampling integral is defined as follows:

[0049]

[0050] Among them, Ω PR It is the subspace obtained by resampling the original integral space Ω. v(x) is the value of the original function (the function before integration) at the query point x, and v(y) is the value of the original function at the key point y, which is the result of integrating the original function. K() is the integral kernel function. K(v(x), v(y)) is used to measure the degree of correlation between v(x) and v(y), which is reflected as a weight in the formula. K(v(x), v(y) i )) is used to measure v(x) and v(y i ) is represented by a weight in the formula, v(y i ) is the discretization representation of v(y), which is determined by the specific discretization degree of the integral space, specifically referring to the original function at the i-th key point (i.e. discrete point) y i The value at , l refers to the subspace Ω PR The starting point in , and l∈[1,n v -m v ],m v is space Ω PR The number of all discrete points, l+m v Refers to the subspace Ω PR The purpose of resampling is to reduce the computational burden of the model without affecting the performance and speed up the model training efficiency. The definition of the Galilean integral is as follows:

[0051]

[0052] Among them, n v Refers to the end point in the original integral space Ω. The progressive resampling is reflected in the point m associated with the progressive resampling integral in the cubic kernel integral v Gradually increase, as the preferred points, are: 4, 16 and 64. During processing, function v0 is added to function v0 after a kernel integration to obtain function v1, function v1 is added to function v1 after a kernel integration to obtain function v2, and function v2 is added to function v2 after a kernel integration to obtain function v3.

[0053] Step S43: The deep supervision module connects the function v0 and the functions v1, v2 and v3 obtained by the three-core kernel integration in the spectral dimension to obtain the function v deep, the size is H×W×(lifting_dim*4).

[0054] Step S44: function v deep Perform spectral inverse linear transform Adjust the number of spectra to obtain the function u, whose size is H×W×S.

[0055] Step S5: Add the obtained function u to the upsampled hyperspectral image Up-LrHSI in step S2 point by point to obtain a high-resolution hyperspectral target image HrHSI.

[0056] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0057] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A spectral image fusion method based on neural operators and progressive resampling, characterized in that: The spectral image fusion method based on neural operators and progressive resampling includes: Step S1: Obtain a low-resolution hyperspectral image of the same target and a high-resolution multispectral image ; Step S2: low-resolution hyperspectral image Perform bilinear interpolation upsampling to obtain high-resolution multispectral images Upsampled hyperspectral images with consistent spatial size , upsample the hyperspectral image and high-resolution multispectral images Input to the multimodal multi-level module, the multimodal multi-level module outputs corresponding to the upsampled hyperspectral image and high-resolution multispectral images Three copies of different sizes 、 、 and 、 、 ; Step S3: Replicas of different sizes 、 、 and 、 、 Perform feature extraction separately and connect the obtained features in the spectral dimension as the initial spectral image function ; Step S4: Initial spectral image function Input to the spectrum fusion neural operator module, the spectrum fusion neural operator module performs the following operations: the initial spectrum image function Perform spectral linear transformation to obtain the function , for the function Perform three kernel integrations step by step to obtain the function 、 and , specifically the function After a kernel integration and the function Add to get the function ,function After a kernel integration and the function Add to get the function ,function After a kernel integration and the function Add to get the function , using the deep supervision module to aggregate 、 、 and Four functions and perform spectral linear inverse transformation to obtain the function ; Each kernel integral includes an asymptotic resampling integral and a Galikin integral; The discrete form of the progressive resampling integral is defined as follows: ； The definition of the Galigin integral is as follows: ； in, is the original integral space The subspace obtained after resampling is The original function at the query point The value of The original function at the key point The value of is the integral kernel function, for The discretization representation of the original function is Key points The value of For subspace The starting point in , For subspace The number of all discrete points, For subspace The end point in is the original integral space The end point in Step S5: Get the function With the upsampled hyperspectral image in step S2 Add together to obtain a high-resolution hyperspectral target image .

2. The spectral image fusion method based on neural operators and progressive resampling according to claim 1, characterized in that: In step S2, the multimodal multi-level module outputs the upsampled hyperspectral images and high-resolution multispectral images Three copies of different sizes 、 、 and 、 、 ,include: Let the upsampled hyperspectral image and high-resolution multispectral images The sizes are and ,in and are the height and width of the spectral image, and Upsampled hyperspectral images and high-resolution multispectral images The number of spectra; Upsample the hyperspectral image As a copy , the size is , the number of groups is , then perform the first replication and group the replication results on the spectral dimension to obtain replicas , the size is ,in is the number of groups, is the number of spectra in each group; for the copies Perform a second replication and group the replication results on the spectral dimension to obtain copies , the size is ,in is the number of groups, is the number of spectra in each group; the high-resolution multispectral image As a copy , the size is , and then perform the first double downsampling in space to obtain a copy , the size is , then based on the copy Perform a second downsampling in the space to obtain a copy , the size is .

3. The spectral image fusion method based on neural operators and progressive resampling according to claim 1, characterized in that: In step S3, the copies of different scales are 、 、 and 、 、 Perform feature extraction separately and connect the obtained features in the spectral dimension as the initial spectral image function ,include: For copies and copies grouped by spectral dimension and Perform group convolution to extract spectral features respectively, and connect the extracted spectral features in the spectral dimension to obtain features ; For copies Perform 3×3 convolution with a kernel size of 3 to extract spatial features, and and Perform 3×3 convolution with a kernel size of 3 to extract spatial features and then perform spatial upsampling to and copy The spatial size is consistent; the three spatial features obtained by channel attention processing are adjusted in the channel dimension by adjusting the weight ratio of each spatial feature, and the three spatial features after adjusting the weight ratio are connected in the spectral dimension to obtain the feature ; The features and features Connect in the spectral dimension to obtain the initial spectral image function .

4. The spectral image fusion method based on neural operators and progressive resampling according to claim 1, characterized in that: In step S5, the obtained function With the upsampled hyperspectral image in step S2 The addition is performed point by point.

Images