A multispectral image fusion based on graph neural network

Abstract

The application belongs to the field of image fusion, and discloses a multispectral image fusion method based on a graph neural network, which comprises the following steps: first, multispectral images and panchromatic images are acquired; pixel features of the multispectral images are extracted by using a convolution network; the multispectral images are subjected to dimension reduction and feature extraction; three-dimensional graph structures of the multispectral images are extracted by a graph embedding method and fused; a heterogeneous graph of multisource features is obtained; spatial features of the graph data are extracted by using a space-time graph convolution; then, the acquired pixel features and spatial features are aggregated by a gating mechanism to obtain the weight of the output features; the multispectral feature map of the final fused spatial features and pixel features is obtained from the weight; the panchromatic images are subjected to the same convolution network; the acquired feature map and the multispectral feature map are fused by an attention mechanism to obtain the fused multispectral images; and the method has higher resolution of the multispectral images.

Description

Technical Field

[0001] This invention relates to the field of image fusion, and more specifically to a multispectral image fusion method based on graph neural networks. Background Technology

[0002] Image fusion technology utilizes image data from the same scene acquired by different sensors, combining and processing them to generate a new, high-quality image. With the rapid development of satellite sensor technology, multispectral imagery is widely used in military systems and environmental analysis. However, due to limitations in satellite sensor technology, only panchromatic images with high spatial resolution but low spectral resolution, or multispectral images with rich spectral information but low spatial resolution, can be acquired. To obtain high spatial resolution multispectral images, remote sensing image fusion technology has become a research hotspot, capable of fusing multispectral and panchromatic images.

[0003] Current remote sensing image fusion techniques can be broadly categorized into four types: component substitution, multiresolution analysis, model-based methods, and deep learning-based methods. Component substitution decomposes a multispectral image into multiple components and replaces some of these components with the spatial components of a panchromatic image. However, due to incomplete component separation, some spectral information may be lost from the multispectral image. Multiresolution analysis injects high-frequency information from the panchromatic image into the multispectral image in the transform domain. Multiresolution analysis better preserves spectral information but sometimes introduces spatial distortion. Model-based methods achieve fusion by establishing optimization models and prior constraints, but their high computational cost and difficulty in selecting optimal manual parameters limit their practical application.

[0004] While current methods using convolutional neural networks to learn spectral and spatial details from remote sensing images have achieved good fusion performance, they still fail to consider the correlation between feature maps of multispectral and hyperspectral images, neglecting the complementary features that may exist between them. This results in weak feature interaction between multispectral and hyperspectral images, low feature extraction accuracy, and consequently, poor image quality. Therefore, designing a network to explore the cross-modal correlation between panchromatic and multispectral images, and to better transfer the spatial texture details of panchromatic images to multispectral images to obtain multispectral images with rich texture information and minimal spectral distortion, is an important problem. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multispectral image fusion method based on graph neural networks. The image fusion method can effectively fuse multispectral images and panchromatic images, and the fused images have strong feature interactions and high image quality.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0007] A multispectral image fusion method based on graph neural networks, characterized by the following steps:

[0008] (S1) Acquire the multispectral image and the corresponding panchromatic image over a period of time;

[0009] (S2) First, extract pixel features from the multispectral image and the panchromatic image using a shared encoder network;

[0010] (S3) The multispectral image is then dimensionality reduced and features are extracted. The graph structures of the three dimensions of the multispectral image are extracted and fused by graph embedding to obtain a heterogeneous graph with multi-source features.

[0011] (S4) Extract features from the acquired heterogeneous graph using spatiotemporal graph convolution to obtain the spatial features of the graph data;

[0012] (S5) The obtained pixel features and spatial features are aggregated through a gating mechanism, and the weights of the features are output. The final multispectral feature map of the fused spatial features and pixel features is obtained from the weights.

[0013] (S6) The feature map of the panchromatic image is obtained and the multispectral feature map of the fused spatial features and pixel features are fused through an attention mechanism;

[0014] (S7) The fused feature map is decoded to obtain the fused multispectral image.

[0015] Preferably, in step (S1), the multispectral camera in step S1 is an imaging camera capable of simultaneously acquiring three or more spectral bands.

[0016] Preferably, in step (S2), the encoder network structure consists of two branches: an upper network for extracting shallow features of the image, which consists of four 3×3 convolutional layers, with the last layer removed and each layer followed by a ReLU activation function; and a lower network for extracting deep features of the image, which first passes through a 1×1 convolutional layer and then is followed by four 3×3 convolutional layers. This convolutional module uses a Nest-based connection method, which can retain more information and obtain deep features. Finally, the feature maps obtained by the upper and lower networks are concatenated.

[0017] Preferably, in step (S3), the acquisition of the graph structure and the acquisition of the three feature maps are as follows: using the dimensionality-reduced spectral data, the physical feature map of the spectral data is extracted by combining infrared spectral features; using linear iterative clustering to determine the superpixel neighbor node information, constructing the edge connection relationship between nodes based on the spatial connectivity relationship of the superpixels, and extracting the spatial feature map; combining the spectral feature similarity of the target, sampling and recombining from different spectral band dimensions to obtain the target spectral feature distribution, and using GNN to effectively represent the spectral data residing on the smooth manifold.

[0018] Preferably, in step (S3), the heterogeneous graph acquisition involves connecting the feature maps of three different node types in three dimensions using a graph autoencoder and employing a graph pooling method based on self-attention to obtain a heterogeneous graph that fuses multi-source features. The autoencoder includes, but is not limited to, graph convolutional autoencoders, variational graph convolutional autoencoders, and adversarial regularized graph autoencoders.

[0019] Preferably, in step (S4), the spatiotemporal graph convolution uses different methods to extract features in the temporal and spatial dimensions. The networks used to extract the temporal dimension include, but are not limited to, RNN, GRU, LSTM, TCN, and Transformer. The networks used to extract the spatial dimension include, but are not limited to, GCN, GAT, and GCN combined with GAT.

[0020] Preferably, in step (S5), the acquisition of the fused feature map involves aggregating two feature maps, using two interconnected fully connected networks. The next step is to pass the aggregated features through an activation function, restricting the function to the range [0,1]. This value represents how much information can pass through the gate, where 0 means no information is allowed to pass and 1 means all information is allowed to pass. The weight of the output feature can be obtained from the gate value, and the weight is multiplied by the pixel feature to obtain the final fused spatiotemporal feature and pixel feature feature map.

[0021] Preferably, in step (S6), the attention mechanism is to combine spatial attention and channel attention to fuse the pixel features of the panchromatic image with the multispectral image features that have been fused with spatial and pixel features.

[0022] Preferably, in step (S7), the decoder is upsampled by four DB modules, each DB module consisting of a 3×3 convolution and a 1×1 convolution. Each DB module adopts the DenseNet connection method, and finally, two 3×3 convolutional layers are used to output the fused multispectral image.

[0023] Compared with the prior art, the present invention provides a multispectral image fusion method based on graph neural networks, which has the following advantages:

[0024] 1. Multispectral images contain rich spectral information, while panchromatic images offer higher spatial resolution. By utilizing graph neural network fusion methods to fully leverage the information from both multispectral and panchromatic images, their information can be effectively combined. This allows for the full utilization of the advantages of both, improving image quality and detail reproduction.

[0025] 2. Graph neural networks learn from graph-structured data and possess excellent global feature learning capabilities. Therefore, fusing multispectral and panchromatic images using graph neural networks can preserve spectral information and spatial consistency. This is crucial for tasks requiring the preservation of object boundaries and color consistency within an image.

[0026] 3. Multispectral and panchromatic images typically contain a large amount of redundant information, especially in the spectral and spatial dimensions. Graph neural network fusion methods can effectively reduce redundant information and extract the most representative and informative features. This can improve the efficiency of image processing and analysis, and reduce the cost of data storage and transmission. Attached Figure Description

[0027] Figure 1 This is a flowchart of a multispectral image fusion method based on graph neural networks according to the present invention. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0029] See Figure 1 The present invention provides a multispectral image fusion method based on graph neural networks, comprising the following steps:

[0030] (S1) Acquire the multispectral image and the corresponding panchromatic image over a period of time;

[0031] (S2) First, extract pixel features from the multispectral image and the panchromatic image using a shared encoder network;

[0032] (S3) The multispectral image is then dimensionality reduced and features are extracted. The graph structures of the three dimensions of the multispectral image are extracted and fused by graph embedding to obtain a heterogeneous graph with multi-source features.

[0033] (S4) Extract features from the acquired heterogeneous graph using spatiotemporal graph convolution to obtain the spatial features of the graph data;

[0034] (S5) The obtained pixel features and spatial features are aggregated through a gating mechanism, and the weights of the features are output. The final multispectral feature map of the fused spatial features and pixel features is obtained from the weights.

[0035] (S6) The feature map of the panchromatic image is obtained and the multispectral feature map of the fused spatial features and pixel features are fused through an attention mechanism;

[0036] (S7) The fused feature map is decoded to obtain the fused multispectral image.

[0037] See Figure 1 In step (S1), the acquired multispectral image is composed of multispectral images of three bands. There are 1000 multispectral images and 1000 panchromatic images, with a pixel size of 256×256.

[0038] See Figure 1 In step (S2), the encoder network consists of two branches. One branch is the upper network, which is used to extract shallow features of the image. It consists of four convolutional layers with 3×3 kernels and a stride of 1, followed by a ReLU activation function. The other branch is the lower network, which is used to extract deep features of the image. It first passes through a 1×1 convolutional layer, and then connects to four convolutional layers with 3×3 kernels and a stride of 1. This convolutional module adopts a Nest-based connection method, which can retain more information and obtain deep features. Finally, the feature maps obtained by the upper and lower networks are concat.

[0039] See Figure 1 In step (S3), the dimensionality reduction and feature extraction method first utilizes augmented vectors to fuse spectral and spatial information.

[0040] x=(u,v,b1,b2,...,b B )=(x 1 ,x 2 ,...,x B+2 ) T (1)

[0041] Where h(u,v) is a pixel on the image, (b1,b2,b3,b...) B () is a band array.

[0042] augmented vector As training data, after normalization, for any x iIn supervised mode, the same category is classified, and the local neighborhood of the pixel is constructed by the k nearest neighbor algorithm. The similarity classification and feature dimensionality reduction of the local neighborhood spatial and spectral information are performed by manifold learning. Combined with spatial spectral polynomial local or neighborhood embedding, the weight allocation of the similarity of spectral features of different pixels in the local neighborhood is completed. Finally, the low-dimensional nonlinear explicit mapping between multispectral data is established by combining quantile matrix multiplication.

[0043] In this embodiment, images of 4 bands were acquired, so B = 4.

[0044] See Figure 1 In step (S3), the heterogeneous graph is obtained by connecting the feature graphs of three different types of nodes and edges using a graph autoencoder-based network structure. Specifically, for each given graph, the node feature vectors between different graphs are analyzed using cosine similarity analysis, and nodes with high similarity among the three graphs are retained. For the three processed graphs, a graph convolutional network is used to calculate the node representation z of each node, and then the following formula is used:

[0045]

[0046] in It is the predicted probability between linking nodes (i, j), where σ is the Sigmoid activation function. A probability greater than 0.8 is set to link, and a probability less than 0.2 is set to not link. This will result in a new graph after linking the three graphs.

[0047] In addition, the extracted new graph is first subjected to a graph neural network (GCN) convolution operation. The GCN learns the feature representation of each node v∈V, that is, it aggregates the features of the neighboring nodes of each node to obtain the feature representation of node v. For each node v, an attention score z is calculated for each node using a self-attention mechanism. Then, the most important nodes are selected using the topk, and the number of nodes to be retained is determined by the pooling ratio k. Here we set k to 0.5. In this way, an attention-based mask map is obtained. The mask map is multiplied with the graph structure of the original input fused heterogeneous information at the corresponding nodes to obtain the final output map, that is, the heterogeneous map fused with multi-source features.

[0048] See Figure 1In step (S4), the spatiotemporal graph convolutional network in the time dimension is TCmodule, and the network in the time dimension is GCN combined with GAT. TCmodule is a network module used to extract the time dimension. It consists of two expansion initiation layers. One expansion initiation layer is processed by the tanh activation function, which acts as a filter for the input. The other branch input is processed by the sigmoid activation function, which controls the amount of information that the filter can pass to the next module. After passing through the network of the time module, the spatial features are extracted by the GCN layer. Then, the information is transferred between nodes through the GAT graph attention layer to capture the dependencies between nodes, and finally the spatiotemporal features are obtained.

[0049] See Figure 1 In step (S5), the gating mechanism is to first fuse multispectral spatiotemporal features f R And pixel features fPi,j:

[0050]

[0051] Here, we use two interconnected fully connected networks for g(), with the hyperbolic tangent function as the activation function. Next, we use the obtained fused features f as a gating mechanism, that is, we pass the aggregated features through a sigmoid activation function, restricting the function to the range [0,1]. This value represents how much information can pass through the gate, where 0 means no information is allowed, and 1 means all information is allowed. In this network, the gating mechanism controls the importance of each pixel, where 0 means the current pixel is completely useless for image recognition, and 1 means the current pixel is extremely important for image recognition. Therefore, the final output function can be expressed as:

[0052]

[0053] ⊙ represents element-wise multiplication, allowing the gate value to control the weights of the output features, ultimately resulting in the feature vector f. fusion It is a multispectral feature map that combines spatiotemporal information and pixel information.

[0054] See Figure 1 In step (S6), the attention mechanism is a combination of channel attention and spatial attention mechanisms to apply the features F of the panchromatic image. Pi,j and multispectral features f fusion Feature learning is performed in both the channel and spatial dimensions to obtain the importance of each channel and the importance of the spatial region, thus obtaining f. ca Feature maps and f of channel attention sa The feature map of spatial attention, with the feature proportions of the two mechanisms fused, is as follows:

[0055] Fφ =(f sa ×0.4+f ca ×0.6)×0.5 (5)

[0056] Since multispectral images consist of multiple bands, we focus more on feature learning along the channel dimension, ultimately obtaining the feature map F. φ .

[0057] See Figure 1 In step (S7), the decoder is upsampled by four DB modules, each DB module consisting of a 3×3 convolution and a 1×1 convolution with a stride of 1, followed by a ReLU activation function. Each DB module uses the DenseNet connection method, and finally, two 3×3 convolutional layers are added. The resulting feature map F φ The network outputs the final fused high-resolution multispectral image.

[0058] The above are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above content. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A multispectral image fusion method based on graph neural networks, characterized in that, Includes the following steps: S1. Acquire the multispectral image and the corresponding panchromatic image over a period of time; S2. First, the multispectral and panchromatic images are used to extract pixel features using a shared encoder network. The encoder network structure consists of two branches: an upper network for extracting shallow features of the image, consisting of four 3×3 convolutional layers with four kernels, excluding the last layer, with each layer followed by a ReLU activation function; and a lower network for extracting deep features of the image, first passing through a 1×1 convolutional layer, followed by four 3×3 convolutional layers. The convolutional modules use a Nest-based connection method, which can retain more information and obtain deep features. Finally, the feature maps obtained by the upper and lower networks are concatenated according to the feature dimensions. S3. The multispectral image is then subjected to dimensionality reduction and feature extraction. Graph structures in the three dimensions of the multispectral image are extracted and fused using graph embedding to obtain a heterogeneous graph with multi-source features. Specifically, the graph structure is obtained by using the dimensionality-reduced spectral data and combining it with infrared spectral features to extract the physical feature map of the spectral data. Linear iterative clustering is used to determine the neighbor node information of superpixels, and edge connections between nodes are constructed based on the spatial connectivity of superpixels to extract the spatial feature map. Combining the spectral feature similarity of the target, the target spectral feature distribution is obtained by sampling and recombining from different spectral band dimensions. A graph neural network is then used to effectively represent the spectral data residing on a smooth manifold. S4. Extract features from the acquired heterogeneous graph using spatiotemporal graph convolution to obtain the spatial features of the graph data; S5. The acquired pixel features and spatial features are aggregated through a gating mechanism, and the weights of the features are output. The final multispectral feature map that fuses the spatial features and pixel features is obtained from the weights. S6. Feature fusion is performed between the feature map of the panchromatic image and the multispectral feature map that integrates spatial and pixel features through an attention mechanism. S7. The fused feature map is passed through a decoder to obtain the fused multispectral image.

2. The multispectral image fusion method based on graph neural networks according to claim 1, characterized in that, In step S1, the multispectral image is captured by a multispectral camera that simultaneously acquires three or more spectral bands.

3. The multispectral image fusion method based on graph neural networks according to claim 1, characterized in that, In step S3, the heterogeneous graph acquisition involves connecting the feature maps of three different node types in three dimensions using a graph autoencoder and employing a graph pooling method based on self-attention to obtain a heterogeneous graph that fuses multi-source features. The graph autoencoder includes a graph convolutional autoencoder, a variational graph convolutional autoencoder, and an adversarial regularized graph autoencoder.

4. The multispectral image fusion method based on graph neural networks according to claim 1, characterized in that, In step S4, the spatiotemporal graph convolution uses different methods to extract features in the temporal and spatial dimensions. The networks used to extract the temporal dimension include RNN, GRU, LSTM, TCN, and Transformer, while the networks used to extract the spatial dimension include GCN, GAT, and GCN combined with GAT.

5. The multispectral image fusion method based on graph neural networks according to claim 1, characterized in that, In step S6, the attention mechanism is to combine spatial attention and channel attention to fuse the pixel features of the panchromatic image with the multispectral image features that have been fused with spatial and pixel features.

Images