A method and system for panchromatic sharpening based on a space-spectrum binarization neural network

Abstract

The application discloses a kind of panchromatic sharpening method and system based on space-spectrum binarization neural network.The application discusses the feasibility of applying binarization neural network (BNN) to panchromatic sharpening, designs the space-spectrum binarization neural network with space-spectrum binarization convolution (S2B-Conv) as the core component, so as to realize panchromatic sharpening.S2B-Conv includes spectral redistribution mechanism and Gabor space feature amplifier, wherein SRM adopts affine transformation, generates its scale and bias parameters through dynamic learning process;GSFA randomly selects different frequencies and angles within the preset range, so as to better process multi-scale and multi-directional spatial elements.Experiments on public data sets WorldView-2 and GaoFen-2 show that the panchromatic sharpening method proposed by the application can achieve good results.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning and computer vision, and particularly relates to a panchromatic sharpening method and system based on a spatial-spectral binary neural network. Background Technology

[0002] With the increasing demand for Earth observation and monitoring, existing optical satellites (such as GaoFen-2 and QuickBird) can simultaneously record low-resolution multispectral (LR-MS) and high-resolution panchromatic (PAN) images of the same scene. Due to the physical limitations of existing satellite sensors, the recorded LR-MS images typically contain rich spectral features but relatively sparse spatial features, while the corresponding PAN images contain rich spatial features but sparse spectral features. Since remote sensing images with multiple spectral bands and high spatial resolution, i.e., high-resolution multispectral (HR-MS) images, are crucial for practical applications, researchers have proposed panchromatic sharpening technology, which is a technique to obtain HR-MS images by reconstructing spatial spectral characteristics during the fusion of PAN and LR-MS images.

[0003] Traditional panchromatic sharpening methods mainly fall into three categories: component replacement (CS) based methods, multi-resolution analysis (MRA) methods, and variational optimization (VO) based methods. In recent years, deep learning-based methods have also made significant progress. Although deep learning models have significantly improved the performance of panchromatic sharpening tasks compared to traditional methods, they typically require powerful hardware and considerable memory and computing power, which hinders their deployment on resource-constrained satellites.

[0004] To improve the efficiency of deep neural networks, researchers have proposed many network compression techniques, including network quantization, parameter pruning, and knowledge distillation. Among these methods, binarized neural networks (BNNs) have become a promising solution for deploying deep learning models on resource-constrained devices. BNNs binarize both weights and activations to only 1 bit and perform pure logical computations, mainly XNOR and bit counting operations, which makes the network highly efficient. However, directly applying model binarization to panchromatic sharpening algorithms may face some challenges: (1) The spectral density and distribution are different in LR-MS images, and similarly, the spectral density and distribution are different between LR-MS and PAN images. In the process of fusing LR-MS and PAN images, performing the same binarization on the activations of different spectral channels may lead to severe distortion of spectral features. (2) Remote sensing scenes themselves are multi-scale and anisotropic. Using ordinary binary convolutions (such as those used in BNNs) directly has two drawbacks: 1) the receptive field is limited, making it difficult to understand the global structure and resulting in blurred or cluttered outlines; 2) the convolution kernel is isotropic, so it learns an "average" feature response during training, which can easily produce jagged effects or blurred edges. Summary of the Invention

[0005] The purpose of this invention is to solve the above-mentioned problems in the prior art and to provide a panchromatic sharpening method and system based on spatial-spectral binarization neural network (S2BNet), thereby performing feature fusion on PAN and LR-MS images with different spatial and spectral features to obtain HR-MS images, making the model more lightweight. In addition, this method does not include complex calculations such as diffusion operations and attention mechanisms that are difficult to implement on edge devices.

[0006] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0007] In a first aspect, the present invention provides a panchromatic sharpening method based on a spatial-spectral binarization neural network. The specific processing flow is as follows: a panchromatic image and a low-resolution multispectral image are input into a spatial-spectral binarization neural network with U-shaped structural features. The low-resolution multispectral image is upsampled and then concatenated with the panchromatic image. The concatenation result is input into a first two-dimensional convolutional layer to generate shallow features. The shallow features are then processed by the first spatial-spectral binarization convolution to obtain a first intermediate feature. The first intermediate feature is processed sequentially by a first encoder, a second encoder, a bottleneck module, a second decoder, and a first decoder to generate a second intermediate feature. A skip connection is used between the first encoder and the first decoder, and a skip connection is used between the second encoder and the second decoder. Afterward, the second spatial-spectral binarization convolution maps the second intermediate feature to a deep feature. Finally, the sum of the shallow features and the deep features is input into a second two-dimensional convolutional layer to generate a reconstructed high-resolution multispectral image.

[0008] Based on the above scheme, each step can be implemented in the following preferred manner.

[0009] As a preferred embodiment of the first aspect mentioned above, the two encoders have the same structure, with each encoder consisting of a basic block and a downsampling module cascaded together; the two decoders have the same structure, with each decoder consisting of an upsampling module, a fusion reduction block, and a basic block cascaded together; and the bottleneck module is a basic block.

[0010] As a preferred embodiment of the first aspect above, the skip connection between the encoder and the decoder is specifically as follows: the output features of the encoder's basic block and the output features of the corresponding decoder's upsampling module are concatenated and then connected to the decoder's fusion reduction block.

[0011] As a preferred embodiment of the first aspect mentioned above, the specific processing flow in the basic block is as follows: the features input to the basic block are processed sequentially through layer normalization, fusion augmentation block, fusion augmentation block, ReLU activation function, spatial-spectral binarization convolution, ReLU activation function, fusion reduction block, and fusion reduction block. The processed features are then residually connected with the features input to the basic block to obtain the output of the basic block.

[0012] As a preferred embodiment of the first aspect mentioned above, in the fusion augmentation block, the features input to the fusion augmentation block are processed in parallel by two spatial-spectral binarization convolutions, and then the two features obtained after parallel processing are concatenated as the output of the fusion augmentation block; in the fusion decrement block, the features input to the fusion decrement block are first split into channels, and then the split features are processed in parallel by two spatial-spectral binarization convolutions, and finally the two features obtained after parallel processing are added as the output of the fusion decrement block.

[0013] As a preferred embodiment of the first aspect mentioned above, in the downsampling module, the features input to the downsampling module are first subjected to average pooling, and then the pooled features are processed in parallel through two spatial-spectral binarization convolutions. Finally, the two features obtained after parallel processing are concatenated as the output of the downsampling module. In the upsampling module, the features input to the upsampling module are sequentially subjected to bilinear processing and average pooling, and then the pooled features are processed in parallel through two spatial-spectral binarization convolutions. Finally, the two features obtained after parallel processing are added as the output of the upsampling module.

[0014] As a preferred embodiment of the first aspect mentioned above, the specific processing flow in spatial-spectral binarization convolution is as follows: The spectral distribution of the input features is adjusted using a spectral redistribution mechanism to obtain the adjusted features; then, a Gabor spatial feature amplifier is used to generate the original weights. The adjusted features and original weights are each binarized to generate corresponding binarized features and weights; XNOR logic operations and bit-counting operations are sequentially performed on the binarized features and weights to replace the floating-point matrix multiplication operations in two-dimensional convolution. The average absolute value of the original weights is multiplied by the operation result to obtain the resized activation; this activation is processed using the RPReLU function and then summed with the input features to achieve residual connections, thereby obtaining the output features.

[0015] As a preferred embodiment of the first aspect mentioned above, the specific processing flow of the spectral redistribution mechanism is as follows: The input features are sequentially processed through average pooling, flattening, a first fully connected layer, a ReLU activation function, and a second fully connected layer to generate the original scaling factor and the original bias; the original scaling factor is restricted to between 0 and 1 using the Sigmoid activation function to obtain the bounded scaling factor, and the original bias is restricted to between -1 and 1 using the tanh function to obtain the bounded bias; the bounded scaling factor and bias are applied to the input features, and the adjusted distribution features are obtained through affine transformation;

[0016] The specific process for generating the original weights using a Gabor spatial feature amplifier is as follows: First, define a first list containing different frequency values ​​and a second list containing different angle values. For each output channel, randomly select a frequency value from the first list and a random angle value from the second list to generate a Gabor kernel and normalize it as the original weights.

[0017] As a preferred embodiment of the first aspect, the adjusted distribution features are binarized using a 1-bit sign function: when the adjusted distribution features are greater than 0, the binarized features are +1; when the adjusted distribution features are less than or equal to 0, the binarized features are -1. The original weights are binarized using a sign function: when the original weights are greater than 0, the binarized weights are +1; when the original weights are less than or equal to 0, the binarized weights are -1. In a second aspect, the present invention provides a panchromatic sharpening system based on a spatial-spectral binarization neural network, comprising:

[0018] The image acquisition module is used to acquire panchromatic images and low-resolution multispectral images;

[0019] The image reconstruction module is used to input panchromatic images and low-resolution multispectral images into a spatial-spectral binarization neural network with U-shaped structural features. After upsampling the low-resolution multispectral image, it is stitched with the panchromatic image. The stitching result is input into the first two-dimensional convolutional layer to generate shallow features. The shallow features are then processed by the first spatial-spectral binarization convolution to obtain the first intermediate features. The first intermediate features are then processed sequentially by the first encoder, the second encoder, the bottleneck module, the second decoder, and the first decoder to generate the second intermediate features. The first encoder and the first decoder are connected by a skip connection, and the second encoder and the second decoder are connected by a skip connection. After that, the second spatial-spectral binarization convolution maps the second intermediate features to deep features. Finally, the sum of the shallow features and the deep features is input into the second two-dimensional convolutional layer to generate the reconstructed high-resolution multispectral image.

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] This invention proposes a novel spatial-spectral binarization neural network, S2BNet, for panchromatic sharpening based on a BNN. S2BNet is a concise and easily deployed basic model with simple computational operations. This invention designs a basic unit for model binarization—the spatial-spectral binarization convolution S2B-Conv. S2B-Conv utilizes a spectral redistribution mechanism to adaptively adjust the density and distribution of spectral features. Simultaneously, S2B-Conv uses a Gabor spatial feature amplifier to capture multi-scale and multi-directional spatial features; these processing steps enable the method of this invention to achieve better results. Attached Figure Description

[0022] Figure 1 This is a diagram showing the overall structure of the spatial-spectral binary neural network in this invention.

[0023] Figure 2 This is a structural diagram of the spatial-spectral binarized convolution S2B-Conv in this invention;

[0024] Figure 3 This is a structural diagram of the basic block in this invention;

[0025] Figure 4 This is a structural diagram of the upsampling module B-Down and the downsampling module B-Up in this invention;

[0026] Figure 5 This is a flowchart illustrating the overall process of the method of the present invention.

[0027] Figure 6 This is a reconstruction result diagram of the GaoFen-2 case provided in this embodiment;

[0028] Figure 7 This is a reconstruction result diagram of the WorldView-2 case provided in this embodiment;

[0029] Figure 8 This is a system block diagram of the present invention. Detailed Implementation

[0030] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0031] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0032] like Figure 1 As shown, in a preferred embodiment of the present invention, the panchromatic sharpening method based on a spatial-spectral binarization neural network includes the following steps. The specific implementation process will be described in detail below.

[0033] The specific processing flow of the above panchromatic sharpening method is as follows: The panchromatic image and the low-resolution multispectral image are input into a spatial-spectral binarization neural network with U-shaped structural features. The low-resolution multispectral image is upsampled and then stitched with the panchromatic image. The stitching result is input into the first two-dimensional convolutional layer to generate shallow features. The shallow features are then processed by the first spatial-spectral binarization convolution to obtain the first intermediate features. The first intermediate features are processed sequentially by the first encoder, the second encoder, the bottleneck module, the second decoder, and the first decoder to generate the second intermediate features. Skip connections are used between the first encoder and the first decoder, and skip connections are used between the second encoder and the second decoder to reduce information loss during processing. After that, the second spatial-spectral binarization convolution maps the second intermediate features to deep features. Finally, the sum of the shallow features and the deep features is input into the second two-dimensional convolutional layer to generate the reconstructed high-resolution multispectral image.

[0034] It should be noted that the panchromatic sharpening problem is to overcome technological limitations and simultaneously acquire remote sensing images with rich spatial and spectral features. Its input consists of a panchromatic image (PAN) containing rich spatial features and relatively sparse spectral features, and a low-resolution multispectral image (LR-MS) containing rich spectral features and relatively sparse spatial features. The output is a high-resolution multispectral image (HR-MS) with rich spectral and spatial features. Although deep learning-based models have achieved good performance, they often have high computational complexity. Binarization neural networks are a good solution for resource-constrained devices, but binarized panchromatic sharpening models have the following problems: due to the inconsistent spectral distribution of PAN / LR-MS images, binarization leads to severe spectral distortion; commonly used binary convolution kernel functions are difficult to adapt to the multi-scale and anisotropic spatial features of remote sensing objects, resulting in severe contour degradation. Therefore, this invention designs a spatial-spectral binary convolution (S2B-Conv) incorporating a spectral redistribution mechanism (SRM) and a Gabor spatial feature amplifier (GSFA) to improve the panchromatic sharpening effect. Among them, SRM can adjust the distribution of the image in the spectral dimension through a dynamic adaptive learning process; GSFA can learn multi-scale and multi-directional spatial features by randomly selecting different frequencies and angles, thereby better handling multi-scale and multi-directional spatial elements of the image. Based on spatial-spectral binarization convolution, this invention proposes a novel spatial-spectral binarization neural network S2BNet for pancolor sharpening. This network contains a series of S2BConv layers, which can adapt to complex spatial-spectral features before the network binarizes.

[0035] In this embodiment, as Figure 1As shown, the spatial-spectral binarization neural network S2BNet includes two full-precision convolutional layers, two spatial-spectral binarization convolutional layers S2B-Conv, two encoders, one bottleneck module, and two decoders. The network as a whole uses only two two-dimensional convolutional layers; all other convolutions are spatial-spectral binarization convolutions. The two encoders have identical structures, each consisting of a basic block and a downsampling module B-DOWN cascaded together. The two decoders also have identical structures, each consisting of an upsampling module B-UP, a fusion reduction block, and a basic block cascaded together. The bottleneck module is simply a basic block. Figure 3 As shown, the basic block contains several core components: Layer Normalization, Fusion Increase Block, S2B-Conv, Fusion Decrease Block, and ReLU activation function. The composition of S2B-Conv will be described in detail later.

[0036] It should be noted that, in this invention, the specific processing flow in spatial-spectral binarization convolution is as follows: The spectral distribution of the input features is adjusted using a spectral redistribution mechanism to obtain the adjusted features; then, a Gabor spatial feature amplifier is used to generate the original weights. The adjusted features and the original weights are each binarized, resulting in corresponding binarized features and weights; XNOR logic operations and bit-counting operations are sequentially performed on the binarized features and weights to replace the floating-point matrix multiplication operations in two-dimensional convolution. The average absolute value of the original weights is multiplied by the operation result to obtain the resized activation; this activation is processed using the RPReLU function and then summed with the input features to achieve residual connections, thereby obtaining the output features.

[0037] Furthermore, the specific processing flow of the spectral redistribution mechanism is as follows: The input features are processed sequentially through average pooling, flattening, the first fully connected layer, the ReLU activation function, and the second fully connected layer to generate the original scaling factor and the original bias; the original scaling factor is restricted to between 0 and 1 using the Sigmoid activation function to obtain the bounded scaling factor, and the original bias is restricted to between -1 and 1 using the tanh function to obtain the bounded bias; the bounded scaling factor and bias are applied to the input features, and the adjusted distribution features are obtained through affine transformation.

[0038] Furthermore, the adjusted distribution features are binarized using a sign function of a 1-bit value: when the adjusted distribution features are greater than 0, the binarized features are +1; when the adjusted distribution features are less than or equal to 0, the binarized features are -1.

[0039] The scalable hyperbolic tangent function is used to approximate the sign function in backpropagation as follows:

[0040]

[0041] in, This represents an element in the feature after the distribution has been adjusted. This represents an element in the binarized feature; It is a learnable parameter that adaptively adjusts the distance between the tanh function and the sign function.

[0042] Furthermore, the original weights are binarized using a sign function: when the original weight is greater than 0, the binarized weight is +1, and when the original weight is less than or equal to 0, the binarized weight is -1.

[0043] Furthermore, the specific process for generating the original weights using the Gabor spatial feature amplifier is as follows: First, define a first list containing different frequency values ​​and a second list containing different angle values. For each output channel, randomly select a frequency value from the first list and randomly select an angle value from the second list to generate a Gabor kernel and normalize it as the original weights.

[0044] The specific processing flow in spatial-spectral binarization convolution is described in detail below. In this embodiment, analysis of LR-MS images reveals significant differences in the distribution and density of spectral information across different bands. The spectral information of PAN images and LR-MS images also shows significant differences. This inconsistency in spectral information can negatively impact the reconstruction of spectral features, leading to potential errors in the final output. Existing techniques attempt to adjust the distribution of spectral information using affine transformations; however, these techniques primarily employ random initialization methods to generate scaling and bias, which are unsuitable for input data. These methods lack the ability to dynamically adjust parameters based on input variables, potentially resulting in poor performance across different datasets.

[0045] To address this limitation, this invention introduces a spectral redistribution mechanism (SRM). This method generates scaling and bias parameters through a data-driven mechanism, which not only enhances adaptability to different input features but also improves the accuracy of feature reconstruction by learning global features to optimize the redistribution process.

[0046] like Figure 2 As shown, firstly, the input features are adjusted using a spectral redistribution mechanism. The spectral distribution, specifically, the input features Global features are obtained after average pooling and flattening operations. Then global features After processing through the first fully connected layer, the ReLU activation function, and the second fully connected layer, specifically, the global features are first processed... The number of channels from Mapped to And after passing through ReLU, it is removed from Mapped to Subsequently, the generated features are split into two separate features along the channel dimension, each feature having One channel, thus generating the original scaling factor. and original bias .here, and These represent the number of channels and the scaling factor, respectively. and These represent the height and width of the feature, respectively.

[0047] Then, the original scaling factor is reduced using the Sigmoid activation function. By limiting the scaling factor to the range of 0 to 1, we obtain the bounded scaling factor. And use the tanh function to change the original bias By restricting the bias to between -1 and 1, we obtain the bounded bias. Finally, the adjusted distribution features are obtained through affine transformation. :

[0048]

[0049]

[0050]

[0051] in, This represents the Sigmoid activation function; This represents the tanh function.

[0052] Features after adjustment of distribution Binarization is performed using the 1-bit sign function Sign to produce the binarized features. Each element has a value of +1 or -1. The binarization process can be represented as:

[0053]

[0054] Since the sign function is non-differentiable, this invention uses a scalable hyperbolic tangent function to approximate the sign function in backpropagation. The specific process is as described above and will not be repeated here. This can be viewed as the real-valued activation of a pixel location after adjustment by the SRM mechanism, i.e., the feature after distribution adjustment. One of the elements; This can be viewed as the binarized output activated at that location, i.e., the binarized feature. The value of the corresponding element in the middle. or .

[0055] For the original weights This invention also utilizes a sign function to generate binarized weights. In the backpropagation process, a piecewise linear function Clip is specifically used:

[0056]

[0057] Subsequently, this invention replaces the computationally intensive floating-point matrix multiplication operations in two-dimensional convolution with purely logical XNOR and bitcount operations, as shown below:

[0058]

[0059] Compare the above results with the original 32-bit weights. The average absolute values ​​are multiplied to reduce the difference between binarization and full-precision weights, i.e.:

[0060]

[0061]

[0062]

[0063] in, Activation after resizing; It is a scaling factor; For the first The original 32-bit weights of each channel; Represents the L1 norm; is the spatial side length of the convolution kernel. For channel indexing; This refers to the number of channels output by S2B-Conv.

[0064] To reduce information loss caused by binarization, this invention introduces a method for... and The residuals from the summation are joined. However, since the ranges of their values ​​differ significantly, directly combining them using the identity mapping may obscure the information in Y. To address this issue, this invention first... Apply the RPReLU function to adjust its range before summing. This step can be represented as:

[0065]

[0066] in, The output feature, the formula for the RPReLU function is:

[0067]

[0068] in, , , These are learnable parameters.

[0069] In this embodiment, parking lots, open spaces, bare ground, and flat construction sites exhibit very similar spectral characteristics, requiring differentiation through spatial features. However, their spatial shapes and sizes are also similar, and the resolution of LR-MS images is too low. Therefore, a method capable of more accurately extracting spatial textures is needed.

[0070] To address the above issues, this invention designs a Gabor Spatial Feature Amplifier (GSFA), which uses Gabor kernels to initialize the weights of convolutional layers. By randomly selecting different frequencies and angles within a preset range, the Gabor kernel enables subsequent convolutional layers to better handle multi-scale and multi-directional features of the image, facilitating the learning of spatial features. Although Gabor kernels are widely used in image processing to simulate the human visual system's perception of edges and textures, existing research has not explored their application in binary convolutional networks. Specifically, GSFA first defines two lists: one containing different frequency values ​​and the other containing different angle values. For each output channel, a frequency value and an angle value are randomly selected from these lists to generate a Gabor kernel, resulting in different Gabor kernels for different output channels. The Gabor kernel is generated based on the following formula:

[0071]

[0072]

[0073] This embodiment only uses the real part, that is:

[0074]

[0075] in, Represents the Gabor kernel function; and This represents the original coordinates of the convolution kernel grid with the top-left corner of the kernel as the origin; This represents the wavelength of the sinusoidal component within the Gabor kernel function; Indicates the direction of the filter; This represents the phase shift of the sine function; This represents the standard deviation of the Gaussian envelope; The aspect ratio represents the influence of the ellipticity of the Gaussian function. and This indicates that the original coordinates , Rotation Rotate the coordinates after the angle so that the filter's principal direction is aligned with... Alignment; It represents the imaginary unit.

[0076] Each parameter of the Gabor function plays a different role in shaping the characteristics of the filter. Because different parameters are chosen for each output channel, therefore... Each output channel will have The model uses several different Gabor kernels, enabling it to learn spatial features at different orientations and scales. Since the generated Gabor kernels are intended to replace the weights in convolutional layers, they need to be normalized so that their mean and variance are similar to the default weights of the convolutional layers. This ensures that the Gabor kernels are compatible with the initialization schemes commonly used in convolutional neural networks.

[0077] It should be noted that in this invention, the first and second 2D convolutional layers refer to standard 2D convolutional layers in frameworks such as PyTorch. These layers use 32-bit floating-point (full-precision) weights and activation values, rather than the 1-bit binary weights / activations compressed using S2B-Conv. 2D convolution is a standard convolution, with weights and activations being floating-point numbers (e.g., float32), and therefore can be considered a "full-precision convolution." This convolution is not binarized, preserving full precision, and is used for key feature extraction and reconstruction steps at the beginning and end of the network.

[0078] The skip connection between the encoder and the decoder is as follows: the output features of the encoder's basic block and the output features of the corresponding decoder's upsampling module are concatenated and then connected to the decoder's fusion reduction block.

[0079] like Figure 3 As shown, the specific processing flow in the basic block is as follows: The features input to the basic block are processed sequentially through layer normalization, fusion augmentation block, fusion augmentation block, ReLU activation function, spatial-spectral binarization convolution, ReLU activation function, fusion decrementation block, and fusion decrementation block. The processed features are then residually concatenated with the features input to the basic block to obtain the output of the basic block.

[0080] like Figure 3As shown, in the fusion augmentation block, the features input to the fusion augmentation block are processed in parallel by two spatial-spectral binarization convolutions, and then the two features obtained after parallel processing are concatenated as the output of the fusion augmentation block; in the fusion decrement block, the features input to the fusion decrement block are first split into channels, and then the split features are processed in parallel by two spatial-spectral binarization convolutions, and finally the two features obtained after parallel processing are added as the output of the fusion decrement block.

[0081] like Figure 4 As shown, in the downsampling module, the features input to the downsampling module are first subjected to average pooling, and then the pooled features are processed in parallel through two spatial-spectral binarization convolutions. Finally, the two features obtained after parallel processing are concatenated as the output of the downsampling module. In the upsampling module, the features input to the upsampling module are processed sequentially through bilinear processing and average pooling, and then the pooled features are processed in parallel through two spatial-spectral binarization convolutions. Finally, the two features obtained after parallel processing are added as the output of the upsampling module.

[0082] In this embodiment, both the fusion augmentation block and the fusion decrement block use spatial-spectral binarization convolution with a kernel size of 1 to aggregate feature maps and modify channels; the upsampling module B-UP uses bilinear interpolation, and then uses spatial-spectral binarization convolution with a kernel size of 3 to enlarge the feature map and halve the number of channels; while the downsampling module B-DOWN uses spatial-spectral binarization convolution with a kernel size of 3 to shrink the feature map and double the number of channels.

[0083] It should be noted that in this invention, the spatial-spectral binarization neural network uses the L1 loss between the real image and the reconstructed image as the reconstruction loss, thereby enabling the spatial-spectral binarization neural network to generate reconstructed images similar to the corresponding real images. The aforementioned reconstruction loss... The definition is as follows:

[0084]

[0085] in, H represents the real image (i.e., the real high-resolution multispectral image), and H represents the reconstructed image (i.e., the reconstructed high-resolution multispectral image). This represents the L1 norm.

[0086] To better demonstrate the specific implementation and technical effects of the present invention, the panchromatic sharpening method based on spatial-spectral binarization neural network shown in the above steps of the preferred implementation is applied to a specific example below.

[0087] Example

[0088] The full-color sharpening method based on a spatial-spectral binarization neural network used in this embodiment is implemented as described above and will not be repeated here. For ease of description, the method shown in the above steps will be referred to as the method of this invention, and the spatial-spectral binarization neural network used will be denoted as S2BNet.

[0089] The overall process in this embodiment can be divided into three stages: data preprocessing, model training, and HR-MS image generation. Figure 5 As shown.

[0090] 1. Data Preprocessing Stage

[0091] This embodiment uses two four-band datasets acquired by the WorldView-2 and GaoFen-2 sensors for experimental analysis of panchromatic sharpening. Since ground truth (GT) images for training are unavailable, following the Wald protocol, both the PAN and MS images are downsampled to a lower scale, allowing the original MS images to serve as ground truth. Because remote sensing images are very large and difficult to directly input into the neural network, this embodiment cropps these images into smaller patches when creating the training and test sets. On the WorldView-2 dataset, the test images are cropped into 145 patches to form the test set, while the training set consists of 1012 patches randomly cropped from other images. The same processing is then applied to the GaoFen-2 dataset, resulting in 136 test patches and 1036 training patches. During training, this embodiment inputs a PAN image with a size of 128×128×1 and an LR-MS image with a size of 32×32×4 to the model. During testing, the sizes of the PAN and LR-MS images are 128×128×1 and 32×32×4, respectively.

[0092] 2. Model Training

[0093] This invention constructs the proposed S2BNet network using PyTorch and trains it on four NVIDIA RTX A5000 GPUs. The batch size for training is set to 16, and the ADAM optimization algorithm is employed, with the optimizer's momentum coefficient... , The initial learning rate is set to Multiply by 0.85 every 100 training rounds. The total number of training rounds is 1500.

[0094] 3. HR-MS image generation

[0095] Using the images from the test set as input, the trained S2BNet network outputs HR-MS images that simultaneously possess rich spatial and spectral features.

[0096] In addition to conducting experiments on down-resolution examples, in order to evaluate the model's performance and generalization ability in real-world scenarios, this embodiment further applies the pre-trained model obtained from the down-resolution data directly to some full-resolution samples and conducts some experiments on full-resolution examples.

[0097] In this embodiment, the test results for the GaoFen-2 case are as follows: Figure 6 As shown, the test results for the WorldView-2 case are as follows: Figure 7 As shown in the figures, each figure provides a visual comparison between the method of this invention and other binarization methods. The first two rows represent the reconstruction results and corresponding MAE images for the reduced resolution examples, while the last row represents the reconstruction results for the full-resolution test case.

[0098] Analysis of the case study GaoFen-2 reveals that the image generated by the S2BNet model of this invention exhibits clear texture and visually pleasing spectrum, reducing aberrations and artifacts; furthermore, the corresponding MAE plot shows minimal bright spots, indicating a high degree of similarity to the real image. In contrast, other comparison methods either exhibit significant spectral distortion or contain blurred spatial textures and distorted edges.

[0099] Analysis of the WorldView-2 case study reveals that among binarization methods, LCRBNN and BNN exhibit poor reconstruction performance, with high brightness values ​​in their error maps. S2BNet demonstrates the best fusion performance, and its reconstructed image shows the least visual difference in color distribution from the real image, preserving spectral information well. BBCU and BiSRNet follow suit.

[0100] It should also be noted that the pancolor sharpening method based on a spatial-spectral binarization neural network in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a pancolor sharpening system based on a spatial-spectral binarization neural network, corresponding to the pancolor sharpening method based on a spatial-spectral binarization neural network provided in the above embodiments, such as... Figure 8 As shown, it includes:

[0101] The image acquisition module is used to acquire panchromatic images and low-resolution multispectral images;

[0102] The image reconstruction module is used to input panchromatic images and low-resolution multispectral images into a spatial-spectral binarization neural network with U-shaped structural features. After upsampling the low-resolution multispectral image, it is stitched with the panchromatic image. The stitching result is input into the first two-dimensional convolutional layer to generate shallow features. The shallow features are then processed by the first spatial-spectral binarization convolution to obtain the first intermediate features. The first intermediate features are then processed sequentially by the first encoder, the second encoder, the bottleneck module, the second decoder, and the first decoder to generate the second intermediate features. The first encoder and the first decoder are connected by a skip connection, and the second encoder and the second decoder are connected by a skip connection. After that, the second spatial-spectral binarization convolution maps the second intermediate features to deep features. Finally, the sum of the shallow features and the deep features is input into the second two-dimensional convolutional layer to generate the reconstructed high-resolution multispectral image.

[0103] It is understood that the pancolor sharpening method based on a spatial-spectral binarization neural network described in the above steps can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the pancolor sharpening method based on a spatial-spectral binarization neural network provided in the above embodiments. This product includes a computer program / instructions that, when executed by a processor, can implement the pancolor sharpening method based on a spatial-spectral binarization neural network as described in the above embodiments.

[0104] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the panchromatic sharpening method based on spatial-spectral binary neural network provided in the above embodiment, which includes a memory and a processor;

[0105] The memory is used to store computer programs;

[0106] The processor is configured to implement the pan-color sharpening method based on a spatial-spectral binary neural network in the above embodiments when executing the computer program.

[0107] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0108] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the pan-color sharpening method based on spatial-spectral binarization neural network provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can implement the pan-color sharpening method based on spatial-spectral binarization neural network in the above embodiments.

[0109] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0110] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0111] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0112] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A panchromatic sharpening method based on a spatial-spectral binarization neural network, characterized in that, The specific processing flow is as follows: The panchromatic image and the low-resolution multispectral image are input into a spatial-spectral binarization neural network with U-shaped structural features. The low-resolution multispectral image is upsampled and then stitched with the panchromatic image. The stitching result is input into the first two-dimensional convolutional layer to generate shallow features. The shallow features are then processed by the first spatial-spectral binarization convolution to obtain the first intermediate features. The first intermediate features are processed sequentially by the first encoder, the second encoder, the bottleneck module, the second decoder, and the first decoder to generate the second intermediate features. The first encoder and the first decoder are connected by a skip connection, and the second encoder and the second decoder are connected by a skip connection. Then, the second spatial-spectral binarization convolution maps the second intermediate features to deep features. Finally, the sum of the shallow features and the deep features is input into the second two-dimensional convolutional layer to generate the reconstructed high-resolution multispectral image.

2. The panchromatic sharpening method based on a spatial-spectral binarization neural network as described in claim 1, characterized in that, The two encoders have the same structure, each consisting of a basic block and a downsampling module cascaded together; the two decoders have the same structure, each consisting of an upsampling module, a fusion reduction block and a basic block cascaded together; the bottleneck module is a basic block.

3. The panchromatic sharpening method based on a spatial-spectral binary neural network as described in claim 2, characterized in that, The skip connection between the encoder and the decoder is as follows: the output features of the encoder's basic block and the output features of the corresponding decoder's upsampling module are concatenated and then connected to the decoder's fusion reduction block.

4. The panchromatic sharpening method based on a spatial-spectral binarization neural network as described in claim 2, characterized in that, The specific processing flow in the basic block is as follows: The features input to the basic block are processed sequentially through layer normalization, fusion augmentation block, fusion augmentation block, ReLU activation function, spatial-spectral binarization convolution, ReLU activation function, fusion reduction block, and fusion reduction block. The processed features are then residually concatenated with the features input to the basic block to obtain the output of the basic block.

5. The panchromatic sharpening method based on a spatial-spectral binarization neural network as described in claim 4, characterized in that, In the fusion augmentation block, the features input to the fusion augmentation block are processed in parallel through two spatial-spectral binarization convolutions, and then the two features obtained after parallel processing are concatenated as the output of the fusion augmentation block; In the fusion reduction block, the features input to the fusion reduction block are first split into channels, and then the split features are processed in parallel through two spatial-spectral binarization convolutions. Finally, the two features obtained after parallel processing are added together as the output of the fusion reduction block.

6. The panchromatic sharpening method based on a spatial-spectral binary neural network as described in claim 2, characterized in that, In the downsampling module, the features input to the downsampling module are first subjected to average pooling, and then the pooled features are processed in parallel through two spatial-spectral binarization convolutions. Finally, the two features obtained from the parallel processing are concatenated as the output of the downsampling module. In the upsampling module, the features input to the upsampling module are processed sequentially through bilinear processing and average pooling, and then the pooled features are processed in parallel through two spatial-spectral binarization convolutions. Finally, the two features obtained from the parallel processing are added as the output of the upsampling module.

7. The panchromatic sharpening method based on a spatial-spectral binarization neural network as described in claim 1, characterized in that, The specific processing flow in spatial-spectral binarization convolution is as follows: The spectral distribution of the input features is adjusted using a spectral redistribution mechanism to obtain the adjusted features; then, the Gabor spatial feature amplifier is used to generate the original weights, and the adjusted features and the original weights are binarized respectively to generate the corresponding binarized features and weights. The binarized features and weights are sequentially subjected to XNOR logical operations and bit counting operations to replace the floating-point matrix multiplication operations in two-dimensional convolution. The average absolute value of the original weights is multiplied with the operation result to obtain the resized activation. The activation is then processed by the RPReLU function and summed with the input features to achieve residual connection, thereby obtaining the output features.

8. The panchromatic sharpening method based on a spatial-spectral binarization neural network as described in claim 7, characterized in that, The specific processing flow of the spectral redistribution mechanism is as follows: The input features are processed sequentially through average pooling, flattening, the first fully connected layer, the ReLU activation function, and the second fully connected layer to generate the original scaling factor and the original bias; the original scaling factor is restricted to between 0 and 1 using the Sigmoid activation function to obtain the bounded scaling factor, and the original bias is restricted to between -1 and 1 using the tanh function to obtain the bounded bias; By applying a bounded scaling factor and bias to the input features, the adjusted distribution of features is obtained through affine transformation. The specific process for generating the original weights using a Gabor spatial feature amplifier is as follows: First, define a first list containing different frequency values ​​and a second list containing different angle values. For each output channel, randomly select a frequency value from the first list and a random angle value from the second list to generate a Gabor kernel and normalize it as the original weights.

9. The panchromatic sharpening method based on a spatial-spectral binary neural network as described in claim 8, characterized in that, The adjusted distribution features are binarized using a sign function with a 1-bit value: when the adjusted distribution feature is greater than 0, the binarized feature is +1; when the adjusted distribution feature is less than or equal to 0, the binarized feature is -1. The original weights are binarized using a sign function: when the original weight is greater than 0, the binarized weight is +1, and when the original weight is less than or equal to 0, the binarized weight is -1.

10. A panchromatic sharpening system based on a spatial-spectral binarization neural network, characterized in that, include: The image acquisition module is used to acquire panchromatic images and low-resolution multispectral images; The image reconstruction module is used to input panchromatic images and low-resolution multispectral images into a spatial-spectral binarization neural network with U-shaped structural features. After upsampling the low-resolution multispectral image, it is stitched with the panchromatic image. The stitching result is input into the first two-dimensional convolutional layer to generate shallow features. The shallow features are then processed by the first spatial-spectral binarization convolution to obtain the first intermediate features. The first intermediate features are then processed sequentially by the first encoder, the second encoder, the bottleneck module, the second decoder, and the first decoder to generate the second intermediate features. The first encoder and the first decoder are connected by a skip connection, and the second encoder and the second decoder are connected by a skip connection. After that, the second spatial-spectral binarization convolution maps the second intermediate features to deep features. Finally, the sum of the shallow features and the deep features is input into the second two-dimensional convolutional layer to generate the reconstructed high-resolution multispectral image.

Images