Multi-task network system based on waveband perception dynamic convolution and method for panchromatic sharpening of multi-spectral image
By using a multi-task network system based on band-aware dynamic convolution, the problem of inconsistent band numbers in multispectral images is solved, achieving efficient full-color sharpening of multispectral images, improving spatial resolution, enhancing knowledge sharing, and reducing the risk of overfitting.
Patent Information
- Application Number
- CN202311069013.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-23
- Publication Date
- 2025-11-25
- Estimated Expiration
- 2043-08-23
AI Technical Summary
In existing multispectral panchromatic sharpening methods, the inconsistent number of bands in multispectral images makes joint learning difficult, and deep neural networks do not fully utilize the prior knowledge of each band.
A multi-task network system based on band-aware dynamic convolution is adopted, which includes a task-general sub-network and a task-specific sub-network. A multi-task learning framework is constructed through band-aware dynamic convolution, feature extraction module, hierarchical feature fusion module, Transformer and reconstruction module, and joint training is performed using multiple satellite datasets.
It achieves efficient panchromatic sharpening of multispectral images, improves spatial resolution, enhances knowledge sharing between different satellite datasets, reduces the risk of overfitting, and provides better performance and robustness.
Smart Images
Figure CN117237210B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multispectral image processing, specifically relating to a multi-task network system based on band-aware dynamic convolution and a multispectral image panchromatic sharpening method based on a multi-task network. Background Technology
[0002] Spatial resolution is a crucial attribute of remote sensing images, especially multispectral images. Improving the spatial resolution of multispectral images is beneficial for applications such as small target identification and scene classification. However, due to the constraints of the signal-to-noise ratio in remote sensing images, it is difficult for satellites in orbit to directly acquire high-resolution multispectral images. Currently, a feasible method is to utilize simultaneously acquired panchromatic images to enhance the spatial resolution of multispectral images. Therefore, this method is called multispectral image panchromatic sharpening. For example, satellites such as Gaofen-2 and QuickBird acquire panchromatic and multispectral images simultaneously.
[0003] In recent years, artificial intelligence, deep learning, and pattern recognition technologies have developed rapidly. Inspired by recent research findings in these fields, many scholars have proposed methods based on deep neural networks. However, the construction of most existing deep neural networks relies on the number of bands in the input multispectral image. This makes it difficult to share general knowledge across multiple satellite datasets. For example, WorldView-2 and WorldView-3 satellites acquire 8-band multispectral images, while WorldView-4 acquires 4-band multispectral images. Deep neural networks that process 8-band multispectral images typically cannot process 4-band multispectral images. Therefore, it is usually difficult to jointly train deep neural networks on the WorldView-2, WorldView-3, and WorldView-4 datasets simultaneously. Furthermore, each band of a multispectral image has a specific physical meaning within a fixed wavelength range. For example, the infrared band has a certain correlation with thermal radiation. However, current deep neural networks do not fully consider the prior knowledge of each band. Summary of the Invention
[0004] The technical problem to be solved by this invention is:
[0005] To overcome the problem of inconsistent band numbers in multispectral images hindering joint learning in existing multispectral panchromatic sharpening methods, this invention provides a multispectral image panchromatic sharpening method based on a band-aware dynamic convolution multi-task network.
[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:
[0007] A multi-task network based on band-aware dynamic convolution is characterized by comprising a task-general subnetwork and a task-specific subnetwork.
[0008] The general sub-network for the task includes two isomorphic feature extraction modules, a hierarchical feature fusion module, a Transformer, and a reconstruction module. The feature extraction module consists of a band-aware dynamic convolution and three residual blocks based on band-aware dynamic convolution. The hierarchical feature fusion module consists of four fusion operations, each of which first concatenates the input features and then fuses them using a band-aware dynamic convolution. The reconstruction module consists of three upsampling modules and one convolution.
[0009] There are multiple task-specific subnetworks, with one separate task-specific subnetwork for each satellite dataset. Each subnetwork consists of three convolutional layers, with the first two layers using the PreLU activation function and the last layer not using any activation function.
[0010] A further technical solution of the present invention: the band-aware dynamic convolution includes three parts: band encoding, feature transformation, and final decision; the band encoding, feature transformation, and final decision are specifically as follows:
[0011] The band coding encodes each band of the multispectral image into a one-dimensional one-hot vector according to the band number. For the first multispectral image band The encoding formula is as follows:
[0012]
[0013] in, ;
[0014] The feature transformation generates different features by employing convolution operations of different sizes;
[0015] The final decision is determined by a combination of transform features and band coding. The decision-making process first uses max pooling to reduce the feature space dimension and integrate information; then it uses element-wise summation and concatenation operations to integrate transform information and band coding information; finally, it uses a multilayer perceptron to make the final decision on the receptive field size.
[0016] A further technical solution of the present invention: the upsampling module of the reconstruction module first concatenates the input features; then uses transposed convolution to improve spatial resolution; and finally uses the PReLU activation function for nonlinear mapping.
[0017] A further technical solution of the present invention: the kernel sizes of the three convolutional layers of the task-specific sub-network are respectively , , .
[0018] A multispectral image panchromatic sharpening method based on a band-aware dynamic convolution multi-task network, characterized by comprising:
[0019] Acquire panchromatic and multispectral images;
[0020] Construct and train a multi-task network; the construction process includes constructing a task-general sub-network and a task-specific sub-network, and the construction of the task-independent sub-network includes constructing a band-aware dynamic convolution module, a feature extraction module, a hierarchical feature fusion module, a Transformer, and a reconstruction module;
[0021] The panchromatic image and the multispectral image are input into the trained multi-task network to obtain the result of the network's forward propagation; that is, the high-resolution multispectral image.
[0022] A further technical solution of the present invention: the training multi-task network includes:
[0023] Input: Multispectral and panchromatic images acquired by multiple satellites
[0024] (1) Constructing a simulation training dataset
[0025] Step 1: Construct a dataset for each satellite based on the Wald protocol, with each image pair containing a panchromatic image. A multispectral image A high-resolution multispectral image ;
[0026] (2) Backpropagation adjustment parameters
[0027] Step 1: Construct the loss function :
[0028]
[0029] in, Indicates the number of simulated training image pairs; express Paradigm; Parameters representing multi-task networks;
[0030] Step 2: Calculate the optimal multi-task network using the Adam optimization algorithm;
[0031] Output: A well-learned multi-task network.
[0032] A computer system is characterized by comprising: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.
[0033] A computer-readable storage medium is characterized by storing computer-executable instructions, which, when executed, are used to implement the method described above.
[0034] The beneficial effects of this invention are as follows:
[0035] This invention provides a multispectral image panchromatic sharpening method based on a band-aware dynamic convolutional multi-task network. This method trains a deep neural network using a multi-task learning paradigm to achieve better performance than single-task networks. Compared to single-task learning, it utilizes more data collected from different tasks. By doing so, this invention can learn robust and global representations for multiple tasks, thereby achieving better knowledge sharing and a more powerful model between tasks, providing better performance for each individual task while reducing the risk of overfitting. Addressing the issue of inconsistent band numbers in multispectral images across different tasks (satellite data), this invention proposes a multi-task network consisting of a task-general subnetwork and a task-specific subnetwork. The task-general subnetwork learns common knowledge across satellite datasets, fusing multispectral images with different band numbers and panchromatic images in a band-by-band processing manner. The task-specific subnetwork is used to learn the spatial-spectral information of specific satellite multispectral images. Therefore, a separate task-specific subnetwork is required for each satellite dataset.
[0036] This invention fully utilizes multi-satellite data to jointly train a multi-task network, thereby effectively learning common knowledge across satellite datasets. The proposed algorithm uses a multi-task network to jointly learn from multi-satellite data. With more data involved, the multi-task network can learn more robust feature representations for multiple tasks. To achieve joint learning from multi-satellite data, the multi-task network comprises a general task sub-network and multiple task-specific sub-networks. The general task sub-network uses a band-by-band processing approach to process multispectral images in different bands; while each task-specific sub-network learns prior knowledge from specific satellite data. Furthermore, the multi-task network employs a hybrid architecture of convolutional networks and Transformers, inheriting the advantages of both. To improve the performance of convolutional operations, this invention proposes a band-aware dynamic convolution, dynamically adjusting the receptive field of the convolutional operation based on the band and features. Attached Figure Description
[0037] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.
[0038] Figure 1The multi-task network framework of this invention. The figure uses WorldView-2 (WV-2), WorldView-3 (WV-3), and WorldView-4 (WV-4) satellites as examples.
[0039] Figure 2 The general sub-network structure diagram for this invention is shown below. "C=32" indicates that the feature pathway is 32; "FEM" indicates the feature extraction module; "Hierarchical fusion" indicates hierarchical feature fusion; and "Reconstruction" indicates the high-resolution multispectral image reconstruction module.
[0040] Figure 3 This invention presents a diagram of a band-sensing dynamic convolutional structure. "MLP" stands for Multilayer Perceptron.
[0041] Figure 4 The task-specific sub-network structure diagram of this invention. Detailed Implementation
[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0043] This invention provides a multispectral image panchromatic sharpening method based on a multi-task network and band-aware dynamic convolution. The method first constructs and trains a multi-task network, then uses the trained network for inference to achieve multispectral image panchromatic sharpening. During the multi-task network learning phase, multiple simulation datasets are constructed based on the widely used Wald protocol. Then, the Adam algorithm is used to adjust the parameters of the multi-task network. The specific multi-task network framework is as follows: Figure 1 As shown, the general sub-network for the task is as follows: Figure 2 As shown, band-aware dynamic convolution is as follows: Figure 3 As shown, the task-specific subnetwork is as follows: Figure 4 As shown.
[0044] The specific implementation process is as follows:
[0045] Constructing a multi-task network system (1) Constructing a general task sub-network. For example... Figure 2As shown, the general sub-network for the task includes two feature extraction modules, one hierarchical feature fusion module, one Transformer, and one reconstruction module. The feature extraction module and the hierarchical feature fusion module employ band-aware dynamic convolution. Therefore, this invention first constructs a band-aware dynamic convolution; then, based on the band-aware dynamic convolution, it constructs a feature extraction module and a hierarchical feature fusion module to extract and fuse the deep hierarchical features of the input band and panchromatic image; finally, it constructs a reconstruction module to reconstruct the high spatial resolution band.
[0046] Step 1: Construct Band-aware Dynamic Convolution (BDConv). For example... Figure 3 The diagram shows the structure of a band-aware dynamic convolution. This convolution consists of three parts: band encoding, feature transformation, and final decision. The band encoding operation encodes each band of the multispectral image into a unique identifier according to its band number. One-hot vector For example, for the first multispectral image... band The encoding formula is as follows:
[0047]
[0048] in, The feature transformation part generates different features by employing convolution operations of different sizes. For example, in... Figure 3 China adopts respectively and The convolution will input features Transformation into features and .in, , and These represent the height, width, and number of channels of the feature, respectively. Here, "GN" indicates group normalization. The final decision is based on the transformed feature ( , ) and band coding The decision is made jointly. Specifically, the decision-making process first uses Global Average Pooling (GAP) to reduce the dimensionality of the feature space and integrate information; then, it integrates transform information and band coding information using element-wise summation and concatenation operations; finally, it uses a Multilayer Perceptron (MLP) to make the final decision regarding the receptive field size. For example, an MLP can... The information in the middle, thereby adjusting the receptive field to .
[0049] Step 2: Construct the Feature Extraction Module (FEM). For example... Figure 2 As shown, the task-specific general sub-network requires the construction of two isomorphic feature extraction modules. Each feature extraction module consists of one band-aware dynamic convolution and three residual blocks based on band-aware dynamic convolutions. The number of channels in these modules is determined by... Figure 3 As shown, the values are 32, 64, 128, and 256 respectively. The stride of each residual block is 2, meaning that the height and width of the output feature of the residual block are half of the input feature.
[0050] Step 3: Construct a hierarchical feature fusion module. For example... Figure 2 As shown, this module consists of four fusion operations (Merge, M). Each fusion operation first concatenates the input features, and then uses band-aware dynamic convolution to fuse the input features.
[0051] Step 4: Construct the Transformer. This module first processes the input features... Transform into Two-dimensional features Then, the two-dimensional feature vectors are mapped to the feature space using the following formula.
[0052]
[0053] in, As a learnable variable, The location is embedded with variables; finally, 12 Transformer layers are used to process the features one by one. Each Transformer layer consists of a multihead self-attention (MSA) and a multilayer perceptron.
[0054] Step 5: Build the refactoring module. For example... Figure 2 As shown, the reconstruction module consists of three upsampling modules and one... It consists of convolutions (Conv). Each upsampling module first concatenates the input features; then it uses transposed convolution to improve spatial resolution; finally, it uses the PReLU (Parametric Rectified Linear Unit) activation function for non-linear mapping. Therefore, this upsampling module is simply referred to as CTP.
[0055] (2) Construct task-specific subnetworks
[0056] Step 1: Construct a separate mission-specific sub-network for each satellite dataset. This network takes the high-resolution multispectral images synthesized by the mission-general network as input and transforms them into high-resolution multispectral images specific to the satellite. This sub-network consists of three convolutional layers, such as... Figure 4 As shown. The kernel sizes of these three convolutional layers are respectively... , , The first two convolutional layers use the PreLU activation function, while the last convolutional layer does not use any activation function. Furthermore, as... Figure 4 As shown, the plus sign on the far right of the network indicates that the network uses residual learning, thereby improving the efficiency of network fitting.
[0057] Input for training the multi-task network: pairs of multispectral and panchromatic images acquired by multiple satellites.
[0058] (1) Constructing a simulation training dataset
[0059] Step 1: Build a dataset for each satellite based on the Wald protocol. For example, build a total of... Data sets { , , …, }. Among them, the first Datasets Include Each image pair contains one panchromatic image. A multispectral image A high-resolution multispectral image .
[0060] (2) Backpropagation adjustment parameters
[0061] Step 1: Construct the loss function As shown in equation (3); Indicates the number of simulated training image pairs; express Paradigm; Parameters representing multi-task networks;
[0062]
[0063] Step 2: Calculate the optimal multi-task network using the Adam optimization algorithm.
[0064] Output: A well-learned multi-task network.
[0065] Input for fusion of panchromatic and multispectral images: a set of image patches ,in The size is , The size is , , and These represent the height, width, and number of channels of a multispectral image, respectively.
[0066] (1) Obtaining high-resolution multispectral images through forward propagation
[0067] Step 1: Convert the panchromatic image and multispectral images The input is fed into a multi-task network to obtain the result of the network's forward propagation; that is, a high-resolution multispectral image. ;
[0068] Output: High spatial resolution multispectral images .
[0069] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention.
Claims
1. A multi-task network system based on band-aware dynamic convolution, characterized in that, It includes a task-general subnetwork and a task-specific subnetwork. The general sub-network for the task includes two isomorphic feature extraction modules, a hierarchical feature fusion module, a Transformer, and a reconstruction module. The feature extraction module consists of a band-aware dynamic convolution and three residual blocks based on band-aware dynamic convolution. The hierarchical feature fusion module consists of four fusion operations, each of which first concatenates the input features and then fuses them using a band-aware dynamic convolution. The reconstruction module consists of three upsampling modules and one convolution. The band-aware dynamic convolution comprises three parts: band encoding, feature transformation, and final decision; the band encoding, feature transformation, and final decision are specifically as follows: The band coding encodes each band of the multispectral image according to its band number. One-hot vector For the first multispectral image band The encoding formula is as follows: in, ; The feature transformation generates different features by employing convolution operations of different sizes; The final decision is determined by a combination of transform features and band coding. The decision-making process first uses max pooling to reduce the feature space dimension and integrate information; then it uses element-wise summation and concatenation operations to integrate transform information and band coding information; finally, it uses a multilayer perceptron to make the final decision on the receptive field size. There are multiple task-specific subnetworks, with one separate task-specific subnetwork for each satellite dataset. Each subnetwork consists of three convolutional layers, with the first two layers using the PreLU activation function and the last layer not using any activation function.
2. The multi-task network system based on band-aware dynamic convolution according to claim 1, characterized in that, The upsampling module of the reconstruction module first concatenates the input features; then it uses transposed convolution to improve spatial resolution; and finally it uses the PReLU activation function for nonlinear mapping.
3. The multi-task network system based on band-aware dynamic convolution according to claim 1, characterized in that, The kernel sizes of the three convolutional layers of the task-specific sub-network are respectively... , , .
4. A method for panchromatic sharpening of multispectral images based on a multi-task network system with band-aware dynamic convolution, characterized in that... include: Acquire panchromatic and multispectral images; Construct and train the multi-task network based on band-aware dynamic convolution as described in claim 1; the construction process includes constructing a task-general sub-network and a task-specific sub-network, and the construction of the task-independent sub-network includes constructing a band-aware dynamic convolution module, a feature extraction module, a hierarchical feature fusion module, a Transformer, and a reconstruction module; The panchromatic image and the multispectral image are input into the trained multi-task network to obtain the result of the network's forward propagation; that is, the high-resolution multispectral image.
5. The multispectral image panchromatic sharpening method based on a band-aware dynamic convolution multi-task network according to claim 4, characterized in that: The training multi-task network includes: Input: Multispectral and panchromatic images acquired by multiple satellites (1) Constructing a simulation training dataset Step 1: Construct a dataset for each satellite based on the Wald protocol, with each image pair containing a panchromatic image. A multispectral image A high-resolution multispectral image ; (2) Backpropagation adjustment parameters Step 1: Construct the loss function : in, Indicates the number of simulated training image pairs; express Paradigm; Parameters representing multi-task networks; Step 2: Calculate the optimal multi-task network using the Adam optimization algorithm; Output: A well-learned multi-task network.
6. A computer system, characterized in that... include: One or more processors, a computer-readable storage medium, for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 4-5.
7. A computer-readable storage medium, characterized in that... The device stores computer-executable instructions, which, when executed, are used to implement the method described in any one of claims 4-5.