Panchromatic multispectral image fusion method and device

The multispectral image is preprocessed and fused through the adaptive filter network model, which solves the problems of spatial distortion and spectral distortion in remote sensing image fusion, realizes high-quality image fusion, and simplifies the data processing process.

CN114331930BActive Publication Date: 2024-11-22AEROSPACE INFORMATION RES INST CAS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111322487.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-09
Publication Date
2024-11-22
Estimated Expiration
2041-11-09

AI Technical Summary

Technical Problem

The existing remote sensing image fusion technology has problems with spatial distortion and spectral distortion, and requires artificially manufactured training data sets, which cannot fully adapt to resolution scale changes, affecting image quality.

Method used

Adaptive filter network model is used to preprocess multispectral images and fuse them through dynamic generation filters to avoid downscale processing and artificial data set manufacturing, and directly fuse them on high-resolution scales.

Benefits of technology

It realizes high-quality image fusion without spatial distortion and spectral distortion, improves image detail and interpretability, and simplifies data processing flow.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114331930B_ABST
    Figure CN114331930B_ABST
Patent Text Reader

Abstract

The present invention provides a method and device for fusion of full-color and multi-spectral images, wherein the method comprises: obtaining a multi-spectral image and a full-color image to be fused; pre-processing the multi-spectral image and the full-color image to obtain a pre-processed multi-spectral image and a pre-processed full-color image; inputting the pre-processed multi-spectral image into an adaptive filter network model to obtain a simulated full-color image; and determining a fused image according to the simulated full-color image, the pre-processed full-color image, and the pre-processed multi-spectral image. The present invention can dynamically generate a filter according to the content of the input image through the adaptive filtering network adopted, thereby enhancing the adaptability of the filter to the content and better achieving fitting. The fusion result has achieved good visual effects, without obvious spectral distortion and spatial distortion, and has also improved quantitative evaluation indicators.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for fusion of full-color and multi-spectral images. Background Art

[0002] With the development of satellite technology, sensor technology, and information technology, the types of remote sensing sensors have evolved from the original single visible light to the current diversification. With the advent of hyperspectral and radar, information reception and storage have become increasingly convenient, and the acquisition of remote sensing data has become increasingly source-based. Since remote sensing obtains surface information based on the electromagnetic spectrum reflected and radiated by the received objects, the results of surface observations by different sensors and different environmental conditions are different, and the spectral resolution, spatial resolution, and temporal resolution of the images obtained are also different. The energy received by the sensor is the double integration of electromagnetic waves in space and wavelength. Higher spatial and spectral resolution means integration in a smaller space and shorter wavelength, and the energy obtained is weaker, resulting in worse image quality. In order to ensure image quality, when one is improved, the other is usually restricted. Temporal resolution depends on the satellite revisit period, which is affected by the height of the satellite's orbit from the ground. The distance to the ground also restricts the spatial resolution. Therefore, there is still a lot of room for improvement in the current development of remote sensing technology. Limited by the hardware equipment of imaging sensors, the remote sensing images obtained cannot have high spectral resolution, high spatial resolution, and high temporal resolution. It is necessary to use remote sensing image fusion algorithms to fuse multi-source remote sensing data. Satellite data sets often use different band combinations. For example, the panchromatic multispectral camera carried by the Gaofen-1 satellite (GF-1) includes four multispectral bands with 8-meter spatial resolution and one panchromatic band with 2-meter resolution. Combining them in this way can reduce the weight, cost, bandwidth and complexity of the satellite. By combining the 8m low spatial resolution multispectral image and the 2m high spatial resolution panchromatic image with the remote sensing image fusion algorithm, a 2m high spatial resolution multispectral fusion image with the characteristics of the original image can be obtained. Compared with the original multispectral image, the fused image has richer details and the spectrum remains basically unchanged. It takes advantage of the advantages and avoids the disadvantages, provides better interpretation and visual effects, and improves the utilization rate of existing data at a lower cost.

[0003] Remote sensing image fusion can be divided into pixel-level fusion, feature-level fusion, and decision-level fusion in terms of fusion level. According to the data source, it can be divided into panchromatic multispectral fusion (i.e. panchromatic sharpening), visible light infrared fusion, and spatiotemporal fusion. Taking panchromatic sharpening, a common task in remote sensing image fusion, as an example, the current related technical methods have the following problems:

[0004] (1) Traditional methods mostly use linear fitting to find the mapping relationship between two fused images, which is not sufficient and has problems such as spatial distortion and spectral distortion;

[0005] (2) Most methods first artificially create training data sets and true values ​​at the downscaling level, and then use the mapping relationship learned from the downscaling to apply it to higher resolution scale fusion. However, the mapping relationship cannot fully adapt to the change in resolution scale, which affects the quality of the fused image. Summary of the invention

[0006] The present invention provides a panchromatic multispectral image fusion method and device, which are used to solve the defects of the prior art that training data sets need to be artificially created and mapping has spatial distortion, and achieve the goal of eliminating the need for downscaling, thus avoiding the work of artificially creating data sets and spatial distortion during mapping.

[0007] In a first aspect, the present invention provides a panchromatic multispectral image fusion method, comprising:

[0008] Acquire a multispectral image and a panchromatic image to be fused;

[0009] Preprocessing the multispectral image and the panchromatic image to obtain a preprocessed multispectral image and a preprocessed panchromatic image;

[0010] Inputting the preprocessed multispectral image into an adaptive filter network model to obtain a simulated full-color image;

[0011] A fused image is determined according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image.

[0012] Further, according to a panchromatic multispectral image fusion method provided by the present invention, the adaptive filter network model is trained by the following method:

[0013] 101: Acquire an original multispectral image and an original panchromatic image to be trained, wherein the original multispectral image and the original panchromatic image to be trained and the multispectral image and the panchromatic image to be fused are data of panchromatic and multispectral sensors carried by the same satellite;

[0014] 102: preprocessing the original multispectral image and the original panchromatic image to be trained to obtain a preprocessed original multispectral image and a preprocessed original panchromatic image;

[0015] 103: Pairing and trimming the preprocessed original multispectral image and the preprocessed original panchromatic image to obtain a plurality of groups of image block pairs;

[0016] 104: Inputting the multispectral image block in the image block pair into the current adaptive filter network model to obtain a corresponding first simulated full-color image;

[0017] 105: Determine a corresponding mean square error loss function according to the first simulated full-color image and a full-color image block in the corresponding image blocks;

[0018] 106: Determine whether the mean square error loss function is less than a preset value;

[0019] 107: If the mean square error loss function is not less than a preset value, adjust the parameters in the original adaptive filter network model, use the adaptive filter network model after parameter adjustment as the current adaptive filter network model, and execute step 104;

[0020] 108: If the mean square error loss function is less than a preset value, the current adaptive filter network model corresponding to the mean square error loss function is used as the final adaptive filter network model.

[0021] Furthermore, according to a panchromatic multispectral image fusion method provided by the present invention, after determining the fused image according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image, the method further comprises:

[0022] The fused image is restored to obtain a final fused image.

[0023] Further, according to a panchromatic multispectral image fusion method provided by the present invention, determining a fused image according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image specifically comprises:

[0024] determining a detail image based on the simulated full-color image and the preprocessed full-color image;

[0025] The detail image is fused with the preprocessed multispectral image to obtain a fused image.

[0026] Furthermore, according to a panchromatic multispectral image fusion method provided by the present invention, the multispectral image is preprocessed, specifically comprising:

[0027] Performing upsampling and interpolation on the multispectral image to obtain an upsampled multispectral image, wherein the upsampling multiple is a resolution ratio of the multispectral image to the panchromatic image;

[0028] The upsampled multispectral image is decentralized to obtain the preprocessed multispectral image.

[0029] Furthermore, according to a panchromatic multispectral image fusion method provided by the present invention, the panchromatic image is preprocessed, specifically comprising:

[0030] The full-color image is subjected to a decentralization process to obtain the pre-processed full-color image.

[0031] In a second aspect, the present invention provides a panchromatic multispectral image fusion device, comprising:

[0032] An image acquisition module, used for acquiring a multispectral image and a panchromatic image to be fused;

[0033] A preprocessing module, used for preprocessing the multispectral image and the panchromatic image to obtain a preprocessed multispectral image and a preprocessed panchromatic image;

[0034] A simulated full-color image acquisition module is used to input the pre-processed multi-spectral image into an adaptive filter network model to obtain a simulated full-color image;

[0035] The fused image module is used to determine a fused image according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image.

[0036] Furthermore, according to a full-color multi-spectral image fusion device provided by the present invention, the simulated full-color image acquisition module is further used for:

[0037] 101: Acquire an original multispectral image and an original panchromatic image to be trained, wherein the original multispectral image and the original panchromatic image to be trained and the multispectral image and the panchromatic image to be fused are data of panchromatic and multispectral sensors carried by the same satellite;

[0038] 102: preprocessing the original multispectral image and the original panchromatic image to be trained to obtain a preprocessed original multispectral image and a preprocessed original panchromatic image;

[0039] 103: Pairing and trimming the preprocessed original multispectral image and the preprocessed original panchromatic image to obtain a plurality of groups of image block pairs;

[0040] 104: Inputting the multispectral image block in the image block pair into the current adaptive filter network model to obtain a corresponding first simulated full-color image;

[0041] 105: Determine a corresponding mean square error loss function according to the first simulated full-color image and a full-color image block in the corresponding image blocks;

[0042] 106: Determine whether the mean square error loss function is less than a preset value;

[0043] 107: If the mean square error loss function is not less than a preset value, adjust the parameters in the original adaptive filter network model, use the adaptive filter network model after parameter adjustment as the current adaptive filter network model, and execute step 104;

[0044] 108: If the mean square error loss function is less than a preset value, the current adaptive filter network model corresponding to the mean square error loss function is used as the final adaptive filter network model.

[0045] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a panchromatic multispectral image fusion method as described in any one of the above are implemented.

[0046] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a panchromatic multispectral image fusion method as described in any one of the above.

[0047] The present invention provides a method and device for fusion of full-color multispectral images, which preprocesses the acquired multispectral image to be fused and the full-color image, and then inputs the preprocessed multispectral image into an adaptive filter network model to obtain a simulated full-color image; then, a fused image is determined based on the simulated full-color image, the preprocessed full-color image, and the preprocessed multispectral image. The adaptive filtering network used in the present invention can dynamically generate a filter according to the content of the input image, enhances the adaptability of the filter to the content, can better achieve fitting, and the fusion result has achieved good visual effects, without obvious spectral distortion and spatial distortion, and has also improved quantitative evaluation indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 It is a schematic diagram of the process of the panchromatic multispectral image fusion method provided by the present invention;

[0050] Figure 2 It is a structural diagram of the adaptive filtering network model provided by the present invention;

[0051] Figure 3 It is a structural schematic diagram of the full-color multi-spectral image fusion device provided by the present invention;

[0052] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Combine the following Figure 1 A panchromatic multispectral image fusion method of the present invention is described, comprising:

[0055] Step 100: Acquire a multispectral image and a panchromatic image to be fused;

[0056] Specifically, satellite data sets often use different band combinations. For example, the panchromatic multispectral camera carried by the Gaofen-1 satellite (GF-1) includes four multispectral bands with 8-meter spatial resolution and one panchromatic band with 2-meter resolution. Combining them in this way can reduce the weight, cost, bandwidth and complexity of the satellite. By combining the 8m low spatial resolution multispectral image and the 2m high spatial resolution panchromatic image with the remote sensing image fusion algorithm, a 2m high spatial resolution multispectral fusion image with the characteristics of the original image can be obtained. The fused image is richer in details than the original multispectral image, and the spectrum remains basically unchanged, which makes the best use of the advantages and avoids the disadvantages, provides better interpretation and visual effects, and improves the utilization rate of existing data at a lower cost. In the embodiment of the present invention, the multispectral image is fused with the panchromatic image, so it is necessary to obtain the multispectral image and the panchromatic image.

[0057] Step 200: preprocessing the multispectral image and the panchromatic image to obtain a preprocessed multispectral image and a preprocessed panchromatic image;

[0058] Specifically, the present invention is carried out on the basis of full resolution, so it is necessary to first upsample and interpolate the low-resolution multispectral image to keep it consistent in size with the high-resolution panchromatic image, and then the interpolated multispectral image and the panchromatic image are both decentralized (i.e., Z-Score normalization) to achieve normalization.

[0059] Step 300: inputting the preprocessed multispectral image into an adaptive filter network model to obtain a simulated full-color image;

[0060] Specifically, the pre-processed multispectral image is input into the adaptive filtering network model, and a simulated full-color image can be output. The adaptive filtering network model is a neural network model that generates a corresponding simulated full-color image according to the multispectral image after training. The multispectral test data is input into the adaptive filter network trained by the simulated full-color image acquisition module in the previous step, and the corresponding separable convolution kernel is generated. The convolution kernel is convolved with the multispectral test data to obtain the corresponding result. The mapping relationship learned by the network is realized by a dynamic filter.

[0061] The network structure of this method is composed of basic convolution units, pooling units, upsampling units, and sub-network units. Each basic convolution unit is composed of three convolution layers and activation layers stacked alternately, the pooling unit is composed of a pooling layer, each upsampling unit is composed of an upsampling layer, a convolution layer, and an activation layer stacked, and the sub-network unit is composed of a basic convolution unit, an upsampling layer, and a convolution layer stacked. The network encoding part of this method is composed of basic convolution units and pooling units stacked alternately, and the decoding part is composed of basic convolution units and upsampling units alternately. Skip-layer connections are added between the encoding and decoding modules to supplement shallow features, and then the sub-network units are connected to generate separable convolution kernel groups. After the training in this method is completed, only the parameters of the network layer are fixed, and the number and size of the dynamic filters generated by the network are fixed, but the content of the filters is not fixed. When testing, according to different input images, the network can generate a set of corresponding horizontal and vertical one-dimensional convolution kernels for each batch, each band, and each pixel point. The convolution kernel itself is related to the input content.

[0062] Step 400: Determine a fused image according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image.

[0063] Specifically, a detail image is determined based on the simulated full-color image and the pre-processed full-color image, and a fused image is determined based on the detail image and the pre-processed multi-spectral image.

[0064] The present invention provides a panchromatic multispectral image fusion method, which preprocesses the acquired multispectral image to be fused and the panchromatic image, and then inputs the preprocessed multispectral image into an adaptive filter network model to obtain a simulated panchromatic image; then, a fused image is determined based on the simulated panchromatic image, the preprocessed panchromatic image, and the preprocessed multispectral image. The adaptive filtering network used in the present invention can dynamically generate a filter according to the content of the input image, enhances the adaptability of the filter to the content, can better achieve fitting, and the fusion result has achieved good visual effects, without obvious spectral distortion and spatial distortion, and has also improved quantitative evaluation indicators.

[0065] Further, according to a panchromatic multispectral image fusion method provided by the present invention, the adaptive filter network model is trained by the following method:

[0066] 101: Acquire an original multispectral image and an original panchromatic image to be trained, wherein the original multispectral image and the original panchromatic image to be trained and the multispectral image and the panchromatic image to be fused are data of panchromatic and multispectral sensors carried by the same satellite;

[0067] 102: preprocessing the original multispectral image and the original panchromatic image to be trained to obtain a preprocessed original multispectral image and a preprocessed original panchromatic image;

[0068] 103: Pairing and trimming the preprocessed original multispectral image and the preprocessed original panchromatic image to obtain a plurality of groups of image block pairs;

[0069] 104: Inputting the multispectral image block in the image block pair into the current adaptive filter network model to obtain a corresponding first simulated full-color image;

[0070] 105: Determine a corresponding mean square error loss function according to the first simulated full-color image and a full-color image block in the corresponding image blocks;

[0071] 106: Determine whether the mean square error loss function is less than a preset value;

[0072] 107: If the mean square error loss function is not less than a preset value, adjust the parameters in the original adaptive filter network model, use the adaptive filter network model after parameter adjustment as the current adaptive filter network model, and execute step 104;

[0073] 108: If the mean square error loss function is less than a preset value, the current adaptive filter network model corresponding to the mean square error loss function is used as the final adaptive filter network model.

[0074] Specifically, combined Figure 2 As shown, in the embodiment of the present invention, the network is built based on the pytorch framework and is mainly composed of four units, including basic convolution units (such as Figure 2 ① in), pooling unit (such as Figure 2 ④ in), upsampling unit (such as Figure 2 ② in), sub-network units (such as Figure 2 Each basic convolution unit (such as Figure 2 ①) is composed of three convolutional layers and activation layers stacked alternately, and the pooling unit (such as Figure 2 ④) in is composed of a pooling layer, each upsampling unit (such as Figure 2②) in the figure is composed of an upsampling layer, a convolutional layer, and an activation layer. The sub-network (such as Figure 2 The network structure of ③ is composed of a basic convolution unit, an upsampling layer, and a convolution layer. In addition, the network structure also adds skip layer connections between some convolution units and upsampling units (such as Figure 2 The output of the convolutional unit on the left side of the skip layer connection is directly added to the output of the upsampling unit on the right side indicated by the arrow, and the result of the sum of the two units is input to the next unit of the network. The two results of the skip layer connection are of the same size. In these units, the convolutional layer size is 3, the stride is 1, and the padding is 1. The number of convolutional layer output channels of the convolutional unit is shown in Figure 2 The annotations under each convolution unit; the pooling layer uses average pooling; the upsampling layer uses bilinear interpolation, and the upsampling multiple is 2; the activation layer uses the ReLU function; the length of the one-dimensional separable convolution kernel corresponding to a single pixel output by the sub-network is 5. The optimizer used by the network is the Adam optimizer. The four bands of multispectral data are stacked as the input of the network. After the input multispectral data passes through a deep network structure stacked with convolutional layers, pooling layers, and upsampling layers, it undergoes the encoding and decoding process. At the same time, skip layer connections are added to supplement shallow features to the network from time to time. The feature maps finally extracted are input into multiple sub-network branches. Each sub-network generates a one-dimensional convolution kernel in one direction of one band, for a total of 4 bands. Each band includes convolution kernels in both horizontal and vertical directions, so a total of 8 sub-networks are set. The entire network is completed by the alternating stacking of basic convolution units and average pooling layers to complete the encoding process, and then the upsampling units and basic convolution units are alternately stacked to complete the decoding process, and finally reach the sub-network unit to output the convolution kernel.

[0075] First, satellite data such as GF-1, WorldView4, IKONOS, and QuickBird are obtained. Good results can be achieved on the scale where the resolution ratio of these multispectral images to panchromatic images is 4. The input channels set in the network are 4 and the output channels are 8, which are adapted to 4-band multispectral images. If more bands of images are to be used for fusion, the number of input and output channels of the network can be adjusted according to the situation. In addition, due to the limitations of hardware equipment and computing memory, there are fewer test images. At least 1024x1024 images can be used for testing, and usually no cropping is required. If the size is larger, appropriate overlapping and cropping of blocks can be considered, and the results of fusion of blocks can be used for image stitching. However, the training involves many parameters and a large amount of data. If the input image size is too large, it may cause insufficient memory. It can be cropped into 256x256 or 128x128 images for training, which can also increase sample diversity.

[0076] Taking the fusion of IKONOS satellite data as an example, the method of using adaptive filtering network for panchromatic sharpening is demonstrated. The multispectral image of IKONOS includes 4 bands with a resolution of 4m, and the resolution of the panchromatic image is 1m, with a resolution ratio of 4. Take 200 pairs of registered 256x256 multispectral images and 1024x1024 panchromatic images, and upsample the low-resolution multispectral image of IKONOS by 4 times according to the resolution ratio to make it the same size as the panchromatic image, both reaching 1024x1024. The bilinear interpolation method can be used. Calculate the mean and variance of the image, perform Z-Score normalization, and complete the normalization. For each training image, 225 images of 128x128 size are cropped sequentially with a step size of 64. Then, a point is randomly selected and used as the upper left corner point of the window to crop 100 images of 128x128 size. The cropping positions of each pair of multispectral images and panchromatic images should correspond one to one, resulting in a training set of 58,500 pairs of image blocks of the same size and position of 180*325. Among them, one pair of images is taken for verification for every 5 pairs, and the rest are used for training.

[0077] Input the multispectral training data into the network (for specific network structure, function and implementation method, see Figure 2 ), generate separable convolution kernels, a total of 4 bands, each pixel in each band has a set of convolution kernels, convolve the obtained convolution kernels with the multispectral training data, and add the convolution results of the 4 bands to obtain a simulated full-color image. Use the full-color training data as the true value, calculate the loss function, use the mean square error to measure the accuracy of the fitting result, calculate the mean square error of the true value full-color image and the simulated full-color image, and optimize and adjust the network according to the loss. The training is completed until the mean square error loss function obtained is less than the preset value.

[0078] The present invention adopts a separable convolution kernel, which reduces the amount of network calculation and improves the speed of network operation, training and learning.

[0079] Furthermore, according to a panchromatic multispectral image fusion method provided by the present invention, after determining the fused image according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image, the method further comprises:

[0080] The fused image is restored to obtain a final fused image.

[0081] Specifically, the fusion result is restored according to the mean and variance of the multispectral test data calculated in the first step, so that it is transformed into the original grayscale range, and the values ​​out of the range are truncated to obtain the final fused image.

[0082] Further, according to a panchromatic multispectral image fusion method provided by the present invention, determining a fused image according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image specifically comprises:

[0083] determining a detail image based on the simulated full-color image and the preprocessed full-color image;

[0084] The detail image is fused with the preprocessed multispectral image to obtain a fused image.

[0085] Specifically, the difference between the simulated high-resolution image and the true value image obtained through the network can obtain a detail image. There are many ways to extract the detail image and fuse the detail image with the low-resolution multispectral image, which can be divided into two types: band-related and band-independent. The method of extracting details by additive modulation is usually independent of the band, and the difference between the simulated image and the true value image is taken as the detail. The method of filling details can be independent of the band or related to the band, depending on the coefficient value when the details are added to each band. The method of filling details by multiplicative modulation is usually related to the band, and the ratio of the simulated image to the true value image is taken as the modulation coefficient, which is multiplied with the low-resolution image. Different modulation methods have different fusion effects, especially when the modulation coefficient is related to the band, it may affect the band spectrum. The present invention is based on the architecture of the traditional IHS transformation idea, and adopts additive modulation to add the difference between the simulated image and the true value image to each band of the multispectral image.

[0086] For example, the multispectral test data is input into the adaptive filter network trained in the previous step to generate the corresponding separable convolution kernel, and the convolution kernel is convolved with the multispectral test data. The convolution results of the four bands are added to generate a simulated full-color image. The full-color test data is subtracted from the simulated full-color image to obtain a detail image, and the detail image is added to each band of the multispectral test data in turn to obtain the result of the fused image.

[0087] Furthermore, according to a panchromatic multispectral image fusion method provided by the present invention, the multispectral image is preprocessed, specifically comprising:

[0088] Performing upsampling and interpolation on the multispectral image to obtain an upsampled multispectral image, wherein the upsampling multiple is a resolution ratio of the multispectral image to the panchromatic image;

[0089] The upsampled multispectral image is decentralized to obtain the preprocessed multispectral image.

[0090] Specifically, the low-resolution multispectral image needs to be upsampled and interpolated first to keep it consistent in size with the high-resolution panchromatic image, and then both the interpolated image and the panchromatic image need to be decentralized (i.e., Z-Score normalization) to achieve normalization.

[0091] Taking the fusion of IKONOS satellite data as an example, the method of using adaptive filtering network for panchromatic sharpening is demonstrated. The multispectral image of IKONOS includes 4 bands with a resolution of 4m, and the resolution of the panchromatic image is 1m, with a resolution ratio of 4. Take 200 pairs of registered 256x256 multispectral images and 1024x1024 panchromatic images, and upsample the low-resolution multispectral image of IKONOS by 4 times according to the resolution ratio to make it the same size as the panchromatic image, both reaching 1024x1024. The bilinear interpolation method can be used. Calculate the mean and variance of the image, perform Z-Score normalization, and complete normalization. Before entering the network, use Z-Score normalization to normalize the multispectral image and the panchromatic image so that the mean of each band is 0 and the variance is 1.

[0092] Furthermore, according to a panchromatic multispectral image fusion method provided by the present invention, the panchromatic image is preprocessed, specifically comprising:

[0093] The full-color image is subjected to a decentralization process to obtain the pre-processed full-color image.

[0094] Specifically, the panchromatic image is normalized by using Z-Score standardization, so that the mean value of each band of the multispectral image and the panchromatic image is 0 and the variance is 1.

[0095] Combination Figure 3 As shown, the present invention provides a full-color multi-spectral image fusion device, comprising:

[0096] An image acquisition module 31 is used to acquire a multispectral image and a panchromatic image to be fused;

[0097] A preprocessing module 32, used for preprocessing the multispectral image and the panchromatic image to obtain a preprocessed multispectral image and a preprocessed panchromatic image;

[0098] A simulated full-color image acquisition module 33 is used to input the pre-processed multi-spectral image into an adaptive filter network model to obtain a simulated full-color image;

[0099] The fused image module 34 is used to determine a fused image according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image.

[0100] Since the device provided in the embodiment of the present invention can be used to execute the method described in the above embodiment, its working principle and beneficial effects are similar, so they are not described in detail here. For specific contents, please refer to the introduction of the above embodiment.

[0101] The present invention provides a panchromatic multispectral image fusion device, which preprocesses the acquired multispectral image to be fused and the panchromatic image, and then inputs the preprocessed multispectral image into an adaptive filter network model to obtain a simulated panchromatic image; then, a fused image is determined according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image. The adaptive filtering network used in the present invention can dynamically generate a filter according to the content of the input image, enhances the adaptability of the filter to the content, can better achieve fitting, and the fusion result has achieved good visual effects, without obvious spectral distortion and spatial distortion, and has also improved quantitative evaluation indicators.

[0102] Furthermore, according to a panchromatic multispectral image fusion device provided by the present invention, the simulated panchromatic image acquisition module 33 is further used for:

[0103] 101: Acquire an original multispectral image and an original panchromatic image to be trained, wherein the original multispectral image and the original panchromatic image to be trained and the multispectral image and the panchromatic image to be fused are data of panchromatic and multispectral sensors carried by the same satellite;

[0104] 102: preprocessing the original multispectral image and the original panchromatic image to be trained to obtain a preprocessed original multispectral image and a preprocessed original panchromatic image;

[0105] 103: Pairing and trimming the preprocessed original multispectral image and the preprocessed original panchromatic image to obtain a plurality of groups of image block pairs;

[0106] 104: Inputting the multispectral image block in the image block pair into the current adaptive filter network model to obtain a corresponding first simulated full-color image;

[0107] 105: Determine a corresponding mean square error loss function according to the first simulated full-color image and a full-color image block in the corresponding image blocks;

[0108] 106: Determine whether the mean square error loss function is less than a preset value;

[0109] 107: If the mean square error loss function is not less than a preset value, adjust the parameters in the original adaptive filter network model, use the adaptive filter network model after parameter adjustment as the current adaptive filter network model, and execute step 104;

[0110] 108: If the mean square error loss function is less than a preset value, the current adaptive filter network model corresponding to the mean square error loss function is used as the final adaptive filter network model.

[0111] Furthermore, according to a panchromatic multispectral image fusion device provided by the present invention, it further includes a restoration module, wherein the restoration module is used to:

[0112] The fused image is restored to obtain a final fused image.

[0113] Further, according to a panchromatic multispectral image fusion device provided by the present invention, the fusion module 34 is specifically used for:

[0114] determining a detail image based on the simulated full-color image and the preprocessed full-color image;

[0115] The detail image is fused with the preprocessed multispectral image to obtain a fused image.

[0116] Further, according to a panchromatic multispectral image fusion device provided by the present invention, the preprocessing module 32 is specifically used for:

[0117] Performing upsampling and interpolation on the multispectral image to obtain an upsampled multispectral image, wherein the upsampling multiple is a resolution ratio of the multispectral image to the panchromatic image;

[0118] The upsampled multispectral image is decentralized to obtain the preprocessed multispectral image.

[0119] Further, according to a panchromatic multispectral image fusion device provided by the present invention, the preprocessing module 32 is specifically used for:

[0120] The full-color image is subjected to a decentralization process to obtain the pre-processed full-color image.

[0121] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a panchromatic multispectral image fusion method, the method comprising: obtaining a multispectral image and a panchromatic image to be fused; preprocessing the multispectral image and the panchromatic image to obtain a preprocessed multispectral image and a preprocessed panchromatic image; inputting the preprocessed multispectral image into an adaptive filter network model to obtain a simulated panchromatic image; and determining a fused image according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image.

[0122] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute a full-color multispectral image fusion method provided by the above methods, the method including: acquiring a multispectral image and a full-color image to be fused; preprocessing the multispectral image and the full-color image to obtain a preprocessed multispectral image and a preprocessed full-color image; inputting the preprocessed multispectral image into an adaptive filter network model to obtain a simulated full-color image; and determining a fused image based on the simulated full-color image, the preprocessed full-color image, and the preprocessed multispectral image.

[0124] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute a panchromatic multispectral image fusion method provided above, the method comprising: acquiring a multispectral image and a panchromatic image to be fused; preprocessing the multispectral image and the panchromatic image to obtain a preprocessed multispectral image and a preprocessed panchromatic image; inputting the preprocessed multispectral image into an adaptive filter network model to obtain a simulated panchromatic image; and determining a fused image based on the simulated panchromatic image, the preprocessed panchromatic image, and the preprocessed multispectral image.

[0125] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A panchromatic multispectral image fusion method, characterized in that: include: Acquire a multispectral image and a panchromatic image to be fused; Preprocessing the multispectral image and the panchromatic image to obtain a preprocessed multispectral image and a preprocessed panchromatic image; Inputting the preprocessed multispectral image into an adaptive filter network model to obtain a simulated full-color image; Determine a fused image according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image; The adaptive filter network model is trained by the following method: 101: Acquire an original multispectral image and an original panchromatic image to be trained, wherein the original multispectral image and the original panchromatic image to be trained and the multispectral image and the panchromatic image to be fused are data of panchromatic and multispectral sensors carried by the same satellite; 102: preprocessing the original multispectral image and the original panchromatic image to be trained to obtain a preprocessed original multispectral image and a preprocessed original panchromatic image; 103: Pairing and trimming the preprocessed original multispectral image and the preprocessed original panchromatic image to obtain a plurality of groups of image block pairs; 104: inputting the multispectral image block in the image block pair into the current adaptive filter network model to obtain a corresponding first simulated full-color image; 105: Determine a corresponding mean square error loss function according to the first simulated full-color image and a full-color image block in the corresponding image blocks; 106: Determine whether the mean square error loss function is less than a preset value; 107: If the mean square error loss function is not less than a preset value, adjust the parameters in the original adaptive filter network model, use the adaptive filter network model after parameter adjustment as the current adaptive filter network model, and execute step 104; 108: If the mean square error loss function is less than a preset value, the current adaptive filter network model corresponding to the mean square error loss function is used as the final adaptive filter network model.

2. The panchromatic multispectral image fusion method according to claim 1, characterized in that: After determining the fused image according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image, the method further includes: The fused image is restored to obtain a final fused image.

3. The panchromatic multispectral image fusion method according to claim 1, characterized in that: Determining a fused image according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image specifically includes: determining a detail image based on the simulated full-color image and the preprocessed full-color image; The detail image is fused with the preprocessed multispectral image to obtain a fused image.

4. The panchromatic multispectral image fusion method according to claim 1, characterized in that: Preprocessing the multispectral image specifically includes: Performing upsampling and interpolation on the multispectral image to obtain an upsampled multispectral image, wherein the upsampling multiple is a resolution ratio of the multispectral image to the panchromatic image; The upsampled multispectral image is decentralized to obtain the preprocessed multispectral image.

5. The panchromatic multispectral image fusion method according to claim 1, characterized in that: Preprocessing the full-color image specifically includes: The full-color image is subjected to a decentralization process to obtain the pre-processed full-color image.

6. A full-color multi-spectral image fusion device, characterized in that: include: An image acquisition module, used for acquiring a multispectral image and a panchromatic image to be fused; A preprocessing module, used for preprocessing the multispectral image and the panchromatic image to obtain a preprocessed multispectral image and a preprocessed panchromatic image; A simulated full-color image acquisition module is used to input the pre-processed multi-spectral image into an adaptive filter network model to obtain a simulated full-color image; A fused image module, used for determining a fused image according to the simulated panchromatic image, the preprocessed panchromatic image and the preprocessed multispectral image; The simulated full-color image acquisition module is also used for: 101: Acquire an original multispectral image and an original panchromatic image to be trained, wherein the original multispectral image and the original panchromatic image to be trained and the multispectral image and the panchromatic image to be fused are data of panchromatic and multispectral sensors carried by the same satellite; 102: preprocessing the original multispectral image and the original panchromatic image to be trained to obtain a preprocessed original multispectral image and a preprocessed original panchromatic image; 103: Pairing and trimming the preprocessed original multispectral image and the preprocessed original panchromatic image to obtain a plurality of groups of image block pairs; 104: inputting the multispectral image block in the image block pair into the current adaptive filter network model to obtain a corresponding first simulated full-color image; 105: Determine a corresponding mean square error loss function according to the first simulated full-color image and a full-color image block in the corresponding image blocks; 106: Determine whether the mean square error loss function is less than a preset value; 107: If the mean square error loss function is not less than a preset value, adjust the parameters in the original adaptive filter network model, use the adaptive filter network model after parameter adjustment as the current adaptive filter network model, and execute step 104; 108: If the mean square error loss function is less than a preset value, the current adaptive filter network model corresponding to the mean square error loss function is used as the final adaptive filter network model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the full-color multispectral image fusion method as described in any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the panchromatic multispectral image fusion method as described in any one of claims 1 to 5 are implemented.